Oct 30 21:51:37.383183 ip-10-0-142-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:51:37.383195 ip-10-0-142-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:51:37.383202 ip-10-0-142-210 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:51:37.383425 ip-10-0-142-210 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:51:47.632782 ip-10-0-142-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:51:47.632803 ip-10-0-142-210 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a057bbe0bccf4e56a21846fd97492341 -- Oct 30 21:53:50.520048 ip-10-0-142-210 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:53:50.843484 ip-10-0-142-210 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:53:50.843484 ip-10-0-142-210 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:53:50.843484 ip-10-0-142-210 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:53:50.843484 ip-10-0-142-210 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:53:50.848160 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848087 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.848160 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848161 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848167 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848170 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848174 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848177 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848179 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848182 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848185 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848187 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848190 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848193 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848195 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848198 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848200 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848203 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848206 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848208 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848214 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848217 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848220 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.848234 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848223 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848225 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848228 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848231 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848233 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848236 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848238 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848241 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848245 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848248 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848250 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848253 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848256 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848258 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848261 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848263 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848266 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848268 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848271 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848274 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.848715 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848277 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848280 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848283 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848286 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848289 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848291 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848294 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848296 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848299 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848301 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848304 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848306 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848309 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848312 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848315 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848318 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848320 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848323 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848326 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.849197 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848328 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848331 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848333 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848336 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848338 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848341 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848343 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848346 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848348 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848351 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848353 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848356 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848359 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848362 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848364 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848367 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848369 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848372 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848374 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848377 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.849643 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848380 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848382 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848385 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848387 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848390 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848392 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848395 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848399 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848401 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848404 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848406 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848410 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848414 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848418 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848421 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848423 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848427 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848429 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848432 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848436 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.850150 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848438 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848441 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848444 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848446 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848449 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.848451 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849221 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849228 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849231 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849234 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849237 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849240 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849243 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849246 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849249 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849251 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849254 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849257 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849260 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849262 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.850636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849265 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849267 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849270 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849273 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849275 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849278 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849281 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849283 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849286 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849289 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849291 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849296 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849300 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849305 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849307 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849311 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849313 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849316 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849318 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.851169 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849321 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849324 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849327 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849330 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849333 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849335 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849354 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849358 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849362 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849364 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849367 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849371 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849374 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849376 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849379 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849382 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849384 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849387 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849390 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.851651 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849392 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849395 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849398 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849401 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849404 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849406 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849409 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849411 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849414 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849416 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849419 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849422 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849425 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849429 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849432 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849435 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849438 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849440 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849443 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.852319 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849446 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849448 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849451 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849454 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849456 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849459 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849462 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849464 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849467 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849470 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849472 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849474 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849477 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849479 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849482 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849484 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849486 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849489 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849491 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849494 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.853119 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849497 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849500 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849502 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849506 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849508 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849511 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849513 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849516 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849518 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849522 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849524 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849527 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849529 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849532 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.849534 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850020 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850025 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850028 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850030 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850033 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.853624 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850036 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850038 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850041 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850044 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850047 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850049 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850052 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850055 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850058 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850060 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850063 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850066 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850069 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850072 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850074 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850077 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850080 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850082 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850085 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.854126 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850088 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850091 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850094 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850097 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850099 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850101 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850104 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850106 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850109 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850112 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850114 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850117 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850119 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850122 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850125 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850127 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850130 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850133 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850135 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850138 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.854636 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850140 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850143 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850146 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850148 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850151 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850153 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850156 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850159 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850161 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850164 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850167 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850169 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850172 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850174 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850180 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850182 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850185 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850187 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850190 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.855494 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850193 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850196 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850199 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850201 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850204 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850206 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850209 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850211 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850214 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850216 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850219 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850221 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850225 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850228 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850231 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850233 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850236 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850238 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850243 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.856285 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850247 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850250 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850253 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850255 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850258 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850261 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850263 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850266 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850270 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850273 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850276 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850279 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850282 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850284 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850287 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850289 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850292 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850295 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850297 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850300 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.856872 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850302 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850305 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850307 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.850311 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851408 2585 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851419 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851426 2585 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851431 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851437 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851441 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851445 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851450 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851453 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851456 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851460 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851464 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851467 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851470 2585 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851473 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851476 2585 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851479 2585 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851482 2585 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851487 2585 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851489 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:53:50.857473 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851493 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851496 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851500 2585 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851503 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851506 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851510 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851513 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851516 2585 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851519 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851522 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851525 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851528 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851532 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851537 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851540 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851543 2585 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851546 2585 flags.go:64] FLAG: --enable-server="true" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851549 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851553 2585 flags.go:64] FLAG: --event-burst="100" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851556 2585 flags.go:64] FLAG: --event-qps="50" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851559 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851563 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851566 2585 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851570 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:53:50.858071 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851573 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851576 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851579 2585 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851582 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851585 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851588 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851591 2585 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851595 2585 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851598 2585 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851601 2585 flags.go:64] FLAG: --feature-gates="" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851605 2585 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851608 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851611 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851614 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851617 2585 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851621 2585 flags.go:64] FLAG: --help="false" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851624 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851628 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851631 2585 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851634 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851637 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851643 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851646 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:53:50.858790 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851649 2585 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851652 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851655 2585 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851658 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851662 2585 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851665 2585 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851668 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851671 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851674 2585 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851677 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851680 2585 flags.go:64] FLAG: --lock-file="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851682 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851686 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851689 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851694 2585 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851697 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851700 2585 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851704 2585 flags.go:64] FLAG: --logging-format="text" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851707 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851710 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851714 2585 flags.go:64] FLAG: --manifest-url="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851717 2585 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851721 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851724 2585 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851728 2585 flags.go:64] FLAG: --max-pods="110" Oct 30 21:53:50.859387 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851731 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851734 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851737 2585 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851740 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851743 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851747 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851750 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851758 2585 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851761 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851777 2585 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851780 2585 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851783 2585 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851786 2585 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851789 2585 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851792 2585 flags.go:64] FLAG: --port="10250" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851795 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851798 2585 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02e6328d3198bef35" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851801 2585 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851804 2585 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851807 2585 flags.go:64] FLAG: --register-node="true" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851809 2585 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851813 2585 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851817 2585 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851820 2585 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851823 2585 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851827 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:53:50.860007 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851830 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851833 2585 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851836 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851839 2585 flags.go:64] FLAG: --runonce="false" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851842 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851845 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851848 2585 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851851 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851854 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851857 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851860 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851864 2585 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851867 2585 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851870 2585 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851873 2585 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851876 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851878 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851881 2585 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851884 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851889 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851892 2585 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851895 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851899 2585 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851901 2585 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851904 2585 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:53:50.860617 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851909 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851911 2585 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851914 2585 flags.go:64] FLAG: --v="2" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851919 2585 flags.go:64] FLAG: --version="false" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851924 2585 flags.go:64] FLAG: --vmodule="" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851927 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.851931 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854847 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854865 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854871 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854875 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854880 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854884 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854889 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854893 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854899 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854903 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854907 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854912 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854916 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854921 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.861224 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854925 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854929 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854933 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854938 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854942 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854946 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854950 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854955 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854961 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854969 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854975 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854980 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854984 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854989 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854994 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.854999 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855005 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855009 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855016 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.861836 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855023 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855029 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855034 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855039 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855044 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855049 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855054 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855058 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855063 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855067 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855071 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855075 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855079 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855084 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855088 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855093 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855097 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855101 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855106 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.862289 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855110 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855114 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855118 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855123 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855127 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855133 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855137 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855141 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855146 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855151 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855155 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855159 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855164 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855168 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855173 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855177 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855181 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855192 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855196 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855201 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.862731 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855205 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855209 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855213 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855217 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855221 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855226 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855230 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855235 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855239 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855245 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855249 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855263 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855269 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855273 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855277 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855281 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855286 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855290 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855295 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.863256 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855299 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855303 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855307 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855312 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855316 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855321 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855325 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855329 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855334 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855338 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855342 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855346 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855350 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855355 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855360 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.855406 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:53:50.863695 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855473 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855480 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855484 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855488 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855492 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855496 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855501 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855505 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855509 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855525 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855531 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855535 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855542 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855548 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855553 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855558 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855562 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855566 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855571 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.864100 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855575 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855579 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855583 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855589 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855596 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855601 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855607 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855612 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855617 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855621 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855626 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855630 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855634 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855638 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855642 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855647 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855651 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855655 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855659 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.864545 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855664 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855668 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855672 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855676 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855688 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855693 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855697 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855702 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855706 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855712 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855716 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855720 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855724 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855729 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855733 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855738 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855742 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855746 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855751 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855755 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.865000 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855759 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855785 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855790 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855795 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855799 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855803 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855807 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855811 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855815 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855819 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855823 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855827 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855831 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855836 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855840 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855844 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855848 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855861 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855865 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855870 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.865480 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855874 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855881 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855886 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855891 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855895 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855900 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855904 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855908 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855913 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855917 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855921 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855925 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855930 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855934 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855939 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855943 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855947 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855951 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855955 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.865962 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855960 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855964 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855969 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855973 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855977 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855981 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855986 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855992 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.855996 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863037 2585 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863055 2585 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863069 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863073 2585 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863128 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863134 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863137 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863140 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863143 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863146 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863148 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.866395 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863151 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863154 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863157 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863160 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863162 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863165 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863168 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863170 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863173 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863175 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863178 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863180 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863183 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863193 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863196 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863199 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863201 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863204 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863207 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863209 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.866878 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863212 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863215 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863219 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863222 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863225 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863228 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863231 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863234 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863236 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863239 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863241 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863244 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863246 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863249 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863251 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863254 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863257 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863260 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863262 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863265 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.867340 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863268 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863271 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863273 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863276 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863278 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863281 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863283 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863286 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863289 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863291 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863294 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863296 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863299 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863302 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863305 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863308 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863311 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863313 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863316 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.867828 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863319 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863321 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863323 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863326 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863328 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863332 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863336 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863339 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863342 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863345 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863348 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863350 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863353 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863355 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863358 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863360 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863363 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863365 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863368 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.868266 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863370 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863373 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863375 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863378 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863380 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863383 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863386 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863388 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863391 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863394 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863399 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863402 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863405 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863408 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863411 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863414 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863417 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863420 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863422 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.868739 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863425 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863427 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863456 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863599 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863604 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863607 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863610 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863613 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863615 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863618 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863621 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863624 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863626 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863629 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863632 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863634 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.869245 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863637 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863639 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863642 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863645 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863647 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863650 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863653 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863656 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863659 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863662 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863664 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863666 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863669 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863672 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863674 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863676 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863679 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863682 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863685 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.869626 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863687 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863690 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863692 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863695 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863698 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863700 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863703 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863706 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863709 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863711 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863715 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863719 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863722 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863724 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863727 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863729 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863732 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863735 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863737 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.870089 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863740 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863743 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863746 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863750 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863754 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863757 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863759 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863762 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863779 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863782 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863785 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863788 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863791 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863793 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863796 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863798 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863801 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863803 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863806 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.870529 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863808 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863811 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863813 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863816 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863818 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863821 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863823 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863826 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863828 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863831 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863833 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863839 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863841 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863844 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863847 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863849 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863852 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863855 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863858 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863861 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.870977 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863863 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863865 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863868 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863871 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863873 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863876 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863879 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863881 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863884 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863886 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863888 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863891 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863894 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863896 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863899 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863901 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.871455 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.863926 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863959 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863963 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863966 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863969 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863971 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863974 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863978 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863981 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863983 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863986 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863988 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863991 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863994 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.863997 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864000 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864003 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864005 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864008 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864010 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:53:50.871839 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864012 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864015 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864017 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864020 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864022 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864025 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864027 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864030 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864033 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864036 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864038 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864041 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864043 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864046 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864048 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864051 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864053 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864056 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864058 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864063 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:53:50.872305 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864066 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864068 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864071 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864073 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864076 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864079 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864081 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864084 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864087 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864089 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864092 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864094 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864097 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864099 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864102 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864105 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864107 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864110 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864113 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864116 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:53:50.872762 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864118 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864121 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864123 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864126 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864128 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864131 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864133 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864135 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864138 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864140 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864143 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864147 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864149 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864152 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864154 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864157 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864159 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864162 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864164 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:53:50.873270 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864167 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864170 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864173 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864176 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864178 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864180 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864183 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864186 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864188 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864191 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864193 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864196 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864199 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864202 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864206 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864209 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864211 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864214 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864216 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:53:50.873717 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864219 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864221 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864223 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864226 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864229 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864233 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864237 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864241 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:50.864245 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.864300 2585 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.865715 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.866528 2585 server.go:1014] "Starting client certificate rotation" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.866621 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.866655 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.868672 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:53:50.874273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.872233 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:53:50.880600 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.880584 2585 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:53:50.883429 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.883415 2585 log.go:25] "Validated CRI v1 image API" Oct 30 21:53:50.884313 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.884288 2585 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:53:50.890260 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.890239 2585 fs.go:135] Filesystem UUIDs: map[44bdeef8-8e21-4464-b53a-f73ca282ede9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e1804596-ffae-45b7-b6f1-88554366a9d5:/dev/nvme0n1p3] Oct 30 21:53:50.890337 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.890259 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:53:50.896268 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.896159 2585 manager.go:217] Machine: {Timestamp:2026-10-30 21:53:50.895112196 +0000 UTC m=+0.277426534 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3007072 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2318c1ed0e34134f87fe622d6f94f2 SystemUUID:ec2318c1-ed0e-3413-4f87-fe622d6f94f2 BootID:a057bbe0-bccf-4e56-a218-46fd97492341 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:4b:ba:f7:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:4b:ba:f7:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:9d:34:1e:4a:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:53:50.896268 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.896265 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:53:50.896397 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.896337 2585 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:53:50.897303 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897280 2585 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:53:50.897434 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897305 2585 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-210.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:53:50.897476 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897436 2585 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:53:50.897476 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897443 2585 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:53:50.897524 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897486 2585 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:53:50.897524 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.897504 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:53:50.898115 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.898106 2585 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:53:50.901868 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.901855 2585 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:53:50.901991 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.901982 2585 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:53:50.902537 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902528 2585 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:53:50.902566 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902541 2585 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:53:50.902566 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902552 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:53:50.902566 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902560 2585 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:53:50.902673 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902568 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:53:50.902993 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.902973 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:53:50.904702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.904684 2585 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:53:50.908909 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.908895 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:53:50.909587 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909576 2585 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:53:50.909627 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909609 2585 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:53:50.909762 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909751 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:53:50.909804 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909785 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:53:50.909804 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909798 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:53:50.909853 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909810 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:53:50.909853 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909818 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:53:50.909853 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.909828 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:53:50.910213 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910204 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:53:50.910244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910215 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:53:50.910244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910225 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:53:50.910244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910231 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:53:50.910244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910240 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:53:50.910386 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910248 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:53:50.910733 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910725 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:53:50.910773 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.910737 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:53:50.914359 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.914342 2585 server.go:1263] "Started kubelet" Oct 30 21:53:50.914447 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.914408 2585 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:53:50.914594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.914490 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:53:50.914715 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.914697 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:53:50.915062 ip-10-0-142-210 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:53:50.916616 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.916593 2585 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:53:50.918523 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.918510 2585 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:53:50.921148 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.921128 2585 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:53:50.921992 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.921963 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:53:50.922358 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.922342 2585 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:53:50.922554 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.922531 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" Oct 30 21:53:50.922969 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.922911 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:53:50.923405 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.923305 2585 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:53:50.923484 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.923413 2585 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:53:50.923648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.923630 2585 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:53:50.923648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.923645 2585 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:53:50.924085 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924066 2585 factory.go:55] Registering systemd factory Oct 30 21:53:50.924151 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924091 2585 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:53:50.924336 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924316 2585 factory.go:153] Registering CRI-O factory Oct 30 21:53:50.924336 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924335 2585 factory.go:223] Registration of the crio container factory successfully Oct 30 21:53:50.924455 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924374 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:53:50.924455 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924399 2585 factory.go:103] Registering Raw factory Oct 30 21:53:50.924455 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.924416 2585 manager.go:1199] Started watching for new ooms in manager Oct 30 21:53:50.925262 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.925242 2585 manager.go:319] Starting recovery of all containers Oct 30 21:53:50.937014 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.936994 2585 manager.go:324] Recovery completed Oct 30 21:53:50.938427 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.938408 2585 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 21:53:50.942427 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.942414 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:50.943910 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.943894 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:50.943966 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.943922 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:50.943966 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.943934 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:50.944414 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.944398 2585 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:53:50.944414 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.944410 2585 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:53:50.944520 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.944428 2585 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:53:50.944520 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.944451 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:50.945218 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.945203 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:50.945266 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.945228 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:50.945266 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.945239 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:50.946623 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.946610 2585 policy_none.go:50] "Start" Oct 30 21:53:50.946672 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.946626 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:53:50.946672 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.946636 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:53:50.948701 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.948686 2585 policy_none.go:44] "Start" Oct 30 21:53:50.985339 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.985301 2585 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:53:50.987714 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.987703 2585 manager.go:341] "Starting Device Plugin manager" Oct 30 21:53:50.987789 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.987734 2585 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:53:50.987789 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.987742 2585 server.go:87] "Starting device plugin registration server" Oct 30 21:53:50.987991 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.987978 2585 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:53:50.988046 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.987994 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:53:50.988146 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.988133 2585 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:53:50.988246 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.988229 2585 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:53:50.988246 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:50.988241 2585 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:53:50.988737 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.988719 2585 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:53:50.988803 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:50.988793 2585 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" Oct 30 21:53:51.038623 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.038594 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:53:51.040784 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.039731 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:53:51.040784 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.039752 2585 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:53:51.040784 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.039787 2585 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:53:51.040784 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.039826 2585 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:53:51.070742 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.070722 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-210.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:53:51.071473 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.070717 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-210.us-west-1.compute.internal.18e36d4799d102ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-210.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-210.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:53:50.914306731 +0000 UTC m=+0.296621069,LastTimestamp:2026-10-30 21:53:50.914306731 +0000 UTC m=+0.296621069,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-210.us-west-1.compute.internal,}" Oct 30 21:53:51.089045 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.089013 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:51.089752 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.089736 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:51.089834 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.089780 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:51.089834 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.089794 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:51.089834 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.089819 2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.140405 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.140364 2585 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal"] Oct 30 21:53:51.140487 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.140437 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:51.141234 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.141219 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:51.141316 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.141250 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:51.141316 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.141265 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:51.143488 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.143473 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:51.143620 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.143605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.143668 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.143634 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:51.144192 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.144178 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:51.144264 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.144201 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:51.144264 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.144214 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:51.145244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.145162 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:51.145244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.145188 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:51.145244 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.145202 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:51.146778 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.146746 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:53:51.146855 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.146837 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhh28" Oct 30 21:53:51.147718 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.147700 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.147826 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.147734 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:53:51.148462 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.148446 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:53:51.148529 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.148476 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:53:51.148529 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.148490 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:53:51.174281 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.174266 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.176951 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.176936 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.225413 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.225391 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.225499 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.225415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.225499 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.225432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.229619 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.229602 2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:51.230990 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.230976 2585 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:51.326193 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.326273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326229 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.326273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.326353 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.326353 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326320 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.326353 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.326339 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.329682 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.329666 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:51.330567 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.330551 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:51.331033 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.331018 2585 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.331110 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.331045 2585 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-210.us-west-1.compute.internal\": node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" Oct 30 21:53:51.423198 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.423150 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.477185 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.477167 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.479323 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.479304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.484982 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.484956 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:53:51.485081 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.485070 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:53:51.543297 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.543279 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:53:51.866455 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.866245 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:53:51.866991 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.866510 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:53:51.866991 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.866510 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:53:51.866991 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.866518 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:53:51.866991 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:51.866522 2585 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2a55f6a5a807453fa6f77ca235301fa-4f791996f38f588f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-210.us-west-1.compute.internal.18e36d479b952531\": read tcp 10.0.142.210:57308->18.116.183.115:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-210.us-west-1.compute.internal.18e36d479b952531 default 5335 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-210.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-210.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-210.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:53:50 +0000 UTC,LastTimestamp:2026-10-30 21:53:51.089798165 +0000 UTC m=+0.472112506,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-210.us-west-1.compute.internal,}" Oct 30 21:53:51.903279 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.903260 2585 apiserver.go:52] "Watching apiserver" Oct 30 21:53:51.923109 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:51.923090 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:53:53.479680 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.479600 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:48:52 +0000 UTC" deadline="2028-04-21 12:48:49.155256078 +0000 UTC" Oct 30 21:53:53.479680 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.479637 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12926h54m55.675622291s" Oct 30 21:53:53.508944 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.507536 2585 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:53:53.508944 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.508302 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-x6pp9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal","openshift-multus/multus-rmh74","openshift-network-diagnostics/network-check-target-g2jrf","kube-system/konnectivity-agent-mv9k2","kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal","openshift-image-registry/node-ca-4hdrj","openshift-ovn-kubernetes/ovnkube-node-4nsbz","kube-system/global-pull-secret-syncer-zrjkd","openshift-multus/network-metrics-daemon-bqpgq","openshift-cluster-node-tuning-operator/tuned-w9glb","openshift-multus/multus-additional-cni-plugins-lgkpc","openshift-network-operator/iptables-alerter-fcl4g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6"] Oct 30 21:53:53.510851 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.510832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.512902 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.512884 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.514977 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.514962 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:53.515055 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.515035 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:53:53.516983 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.516966 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.518974 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.518954 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.521283 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.521266 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.523630 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.523614 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.523714 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.523691 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:53:53.525858 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.525841 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.525936 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.525901 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:53:53.528368 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.528355 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.530860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.530842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.533016 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.533001 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.535102 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.535085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.551765 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.551751 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:53:53.559177 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xrf\" (UniqueName: \"kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559240 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559185 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.559240 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tv8\" (UniqueName: \"kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.559303 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.559355 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559355 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559454 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559454 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559454 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559441 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559610 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.559610 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559505 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.559610 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.559610 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559632 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559667 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx44\" (UniqueName: \"kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.559814 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559868 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559919 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.559999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560030 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560027 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wl4\" (UniqueName: \"kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8d8\" (UniqueName: \"kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560174 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560213 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8426f\" (UniqueName: \"kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560238 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560270 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.560308 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9ww\" (UniqueName: \"kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560326 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560441 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whbs\" (UniqueName: \"kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560496 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560545 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560656 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.560806 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560702 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560718 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560788 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560835 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560982 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.560995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561363 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561043 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561084 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tw6w\" (UniqueName: \"kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561143 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561167 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.561805 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.561194 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.565254 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565233 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:53:53.565338 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565285 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:53:53.565399 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565387 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.565436 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565405 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.565590 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565561 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8jkh4\"" Oct 30 21:53:53.565651 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.565589 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hbp5d\"" Oct 30 21:53:53.566400 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.566379 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.566836 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.566822 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.569085 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.569055 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:53:53.569365 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.569351 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:53:53.569534 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.569516 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjl7c\"" Oct 30 21:53:53.570835 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.570821 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.571314 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.571302 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4dnxg\"" Oct 30 21:53:53.571903 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.571891 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:53:53.571940 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.571893 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.573301 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.573290 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:53:53.573496 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.573485 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pgbrx\"" Oct 30 21:53:53.574260 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.574250 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:53:53.574931 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.574917 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:53:53.575083 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.575069 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.575235 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.575223 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.614076 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.614058 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:53:53.614844 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.614807 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:53:53.614844 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.614811 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.614844 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.614833 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4xfw\"" Oct 30 21:53:53.616220 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.616201 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:53:53.618848 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.616666 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z7lww\"" Oct 30 21:53:53.618848 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.616879 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s9wtv\"" Oct 30 21:53:53.618848 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.618308 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ghldd\"" Oct 30 21:53:53.618848 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.618803 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.620249 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.620234 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:53:53.620327 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.620281 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.620327 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.620290 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.620327 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.620308 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:53:53.621173 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.621160 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:53:53.621291 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.621277 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:53:53.621357 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.621292 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:53:53.621357 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.621334 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cmxsf" Oct 30 21:53:53.661505 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.661594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661509 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.661594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.661594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661545 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661558 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.661594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661588 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whbs\" (UniqueName: \"kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661604 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661618 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661636 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661663 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661679 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661679 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661696 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661706 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661729 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661755 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661797 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.661837 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661897 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661909 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.661952 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.661968 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662007 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.662023 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:53:54.16200586 +0000 UTC m=+2.216512567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662049 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662114 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662135 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.662407 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662162 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662184 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662190 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662215 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662220 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662240 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662254 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662273 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662322 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.663207 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662365 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662377 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662419 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw6w\" (UniqueName: \"kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662443 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662463 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662470 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662506 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xrf\" (UniqueName: \"kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662587 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tv8\" (UniqueName: \"kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.664004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662657 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662685 2585 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662534 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662841 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662833 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662786 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662868 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662887 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.664692 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662709 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662870 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662903 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662935 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662963 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662973 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662975 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.662981 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663009 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx44\" (UniqueName: \"kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663031 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663035 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663069 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663091 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663107 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665181 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663116 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663215 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663221 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663274 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663297 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663307 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663361 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663364 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663489 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wl4\" (UniqueName: \"kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8d8\" (UniqueName: \"kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663543 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663555 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.665702 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663562 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663646 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663683 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663725 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8426f\" (UniqueName: \"kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663730 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663766 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9ww\" (UniqueName: \"kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663811 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663850 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666292 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663886 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663886 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663933 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663930 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.663969 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.664005 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.664018 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.664051 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:54.164037134 +0000 UTC m=+2.218543856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.664157 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.664445 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.666091 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.666163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.666480 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.666860 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.666483 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.725408 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.725389 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whbs\" (UniqueName: \"kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.747811 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.747763 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw6w\" (UniqueName: \"kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.752190 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.752166 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx44\" (UniqueName: \"kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.752331 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.752268 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8426f\" (UniqueName: \"kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.771647 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.771618 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xrf\" (UniqueName: \"kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.814699 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.814673 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:53:53.814699 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.814695 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:53:53.814892 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.814708 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:53.814892 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:53.814768 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:53:54.314750041 +0000 UTC m=+2.369256767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:53.816771 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.816746 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tv8\" (UniqueName: \"kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.816881 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.816752 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9ww\" (UniqueName: \"kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:53.816881 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.816790 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8d8\" (UniqueName: \"kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.817276 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.817260 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wl4\" (UniqueName: \"kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.829197 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.829170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:53:53.850876 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.850858 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rmh74" Oct 30 21:53:53.856397 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.856377 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:53:53.869111 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.869084 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b39b57_4d19_426b_982b_4e197e37736f.slice/crio-b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b WatchSource:0}: Error finding container b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b: Status 404 returned error can't find the container with id b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b Oct 30 21:53:53.869568 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.869547 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81c0006_0d2d_4e09_b222_99dd64093045.slice/crio-a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991 WatchSource:0}: Error finding container a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991: Status 404 returned error can't find the container with id a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991 Oct 30 21:53:53.870666 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.870643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:53:53.870927 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.870909 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63e129e0cfb39daa6732f74d4a5cd8a.slice/crio-339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624 WatchSource:0}: Error finding container 339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624: Status 404 returned error can't find the container with id 339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624 Oct 30 21:53:53.871302 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.871282 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda689496985b50dbcdef398d32e6ac5d5.slice/crio-80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa WatchSource:0}: Error finding container 80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa: Status 404 returned error can't find the container with id 80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa Oct 30 21:53:53.875010 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.874986 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:53:53.880012 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.879989 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56f8924_c137_477d_a859_6defeffe45d4.slice/crio-6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00 WatchSource:0}: Error finding container 6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00: Status 404 returned error can't find the container with id 6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00 Oct 30 21:53:53.906684 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.906662 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:53:53.914745 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.914729 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:53:53.915004 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.914981 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e6cf70_3f1a_40a1_b849_fcea928413c7.slice/crio-8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a WatchSource:0}: Error finding container 8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a: Status 404 returned error can't find the container with id 8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a Oct 30 21:53:53.921721 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.921702 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03a9d9_1567_4254_9747_d9836c323a2c.slice/crio-854b1d0bdefa1d760aa5aaf67218ec44bc72d39af3836604da4d7a9ff31be771 WatchSource:0}: Error finding container 854b1d0bdefa1d760aa5aaf67218ec44bc72d39af3836604da4d7a9ff31be771: Status 404 returned error can't find the container with id 854b1d0bdefa1d760aa5aaf67218ec44bc72d39af3836604da4d7a9ff31be771 Oct 30 21:53:53.932627 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.932604 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:53:53.940472 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.940455 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc5a0a3_e851_4c62_8a13_0985bbb5068e.slice/crio-414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c WatchSource:0}: Error finding container 414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c: Status 404 returned error can't find the container with id 414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c Oct 30 21:53:53.950446 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.950430 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:53:53.958522 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:53.958506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:53:53.958726 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.958710 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6ed4c4_5043_4424_8b2d_826f478062c6.slice/crio-1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837 WatchSource:0}: Error finding container 1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837: Status 404 returned error can't find the container with id 1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837 Oct 30 21:53:53.965614 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:53.965596 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60aa182_3a90_40fd_b14d_620259e35523.slice/crio-65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929 WatchSource:0}: Error finding container 65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929: Status 404 returned error can't find the container with id 65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929 Oct 30 21:53:54.166302 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.166238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:54.166302 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.166269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:54.166424 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.166357 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:54.166424 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.166357 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:54.166424 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.166401 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:53:55.166386166 +0000 UTC m=+3.220892889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:54.166424 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.166414 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:55.166407994 +0000 UTC m=+3.220914700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:54.269728 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.269706 2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:54.367519 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.367493 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:54.367649 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.367635 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:53:54.367690 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.367654 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:53:54.367690 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.367662 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:54.367753 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:54.367707 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:53:55.367691061 +0000 UTC m=+3.422197780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:54.371161 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.371123 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerStarted","Data":"414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c"} Oct 30 21:53:54.372104 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.372085 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mv9k2" event={"ID":"a81c0006-0d2d-4e09-b222-99dd64093045","Type":"ContainerStarted","Data":"a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991"} Oct 30 21:53:54.373068 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.373050 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rmh74" event={"ID":"b1b39b57-4d19-426b-982b-4e197e37736f","Type":"ContainerStarted","Data":"b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b"} Oct 30 21:53:54.373993 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.373977 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerStarted","Data":"80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa"} Oct 30 21:53:54.374917 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.374893 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" event={"ID":"b60aa182-3a90-40fd-b14d-620259e35523","Type":"ContainerStarted","Data":"65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929"} Oct 30 21:53:54.375824 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.375807 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcl4g" event={"ID":"5b6ed4c4-5043-4424-8b2d-826f478062c6","Type":"ContainerStarted","Data":"1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837"} Oct 30 21:53:54.376703 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.376689 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9glb" event={"ID":"cf03a9d9-1567-4254-9747-d9836c323a2c","Type":"ContainerStarted","Data":"854b1d0bdefa1d760aa5aaf67218ec44bc72d39af3836604da4d7a9ff31be771"} Oct 30 21:53:54.377582 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.377551 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a"} Oct 30 21:53:54.378601 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.378568 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4hdrj" event={"ID":"f56f8924-c137-477d-a859-6defeffe45d4","Type":"ContainerStarted","Data":"6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00"} Oct 30 21:53:54.379494 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.379475 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"c63e129e0cfb39daa6732f74d4a5cd8a","Type":"ContainerStarted","Data":"339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624"} Oct 30 21:53:54.590500 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.590440 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:54.624322 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.624268 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:48:53 +0000 UTC" deadline="2028-06-17 04:45:00.137206402 +0000 UTC" Oct 30 21:53:54.624322 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.624315 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14286h51m5.512894536s" Oct 30 21:53:54.653708 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:53:54.653678 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da87eda_28f6_4a7c_a004_42dd0b6a0d58.slice/crio-5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57 WatchSource:0}: Error finding container 5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57: Status 404 returned error can't find the container with id 5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57 Oct 30 21:53:54.836822 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:54.836791 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:53:55.172399 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.172365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:55.172560 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.172423 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:55.172654 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.172605 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:55.172728 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.172713 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:53:57.172687188 +0000 UTC m=+5.227193900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:55.172837 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.172823 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:55.172894 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.172864 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:57.172852176 +0000 UTC m=+5.227358882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:55.368513 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.368480 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:55.369149 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.368632 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:53:55.369149 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.368714 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:55.369149 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.368833 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:53:55.369149 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.368894 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:55.369149 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.368958 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:53:55.374621 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.374380 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:55.374875 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.374562 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:53:55.374875 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.374775 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:53:55.374875 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.374790 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:55.374875 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:55.374847 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:53:57.374827405 +0000 UTC m=+5.429334117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:55.396336 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.396308 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x6pp9" event={"ID":"6da87eda-28f6-4a7c-a004-42dd0b6a0d58","Type":"ContainerStarted","Data":"5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57"} Oct 30 21:53:55.625510 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.625406 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:48:53 +0000 UTC" deadline="2028-07-08 03:25:16.88058827 +0000 UTC" Oct 30 21:53:55.625510 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:55.625439 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14789h31m21.255153457s" Oct 30 21:53:57.190035 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.189951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:57.190514 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.190045 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:57.190514 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.190180 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:57.190514 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.190241 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:01.190223055 +0000 UTC m=+9.244729763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:53:57.190514 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.190307 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:57.190514 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.190342 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:54:01.190330524 +0000 UTC m=+9.244837237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:53:57.368390 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.368356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:57.368547 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.368502 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:53:57.368628 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.368598 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:57.368684 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.368668 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:53:57.368731 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.368720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:57.368807 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.368789 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:53:57.391828 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:57.391541 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:57.391828 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.391698 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:53:57.391828 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.391715 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:53:57.391828 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.391725 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:57.391828 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:57.391784 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:54:01.391756931 +0000 UTC m=+9.446263637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:53:58.429687 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:58.428663 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="63e6b63128b3474c535e2442dbd3a5e30591494e70eaeaa65bcf07c108b38605" exitCode=0 Oct 30 21:53:58.429687 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:58.428718 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"63e6b63128b3474c535e2442dbd3a5e30591494e70eaeaa65bcf07c108b38605"} Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.368237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:59.368624 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.368708 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:59.368782 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.368833 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:53:59.369048 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:53:59.368903 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:53:59.440421 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.439764 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mv9k2" event={"ID":"a81c0006-0d2d-4e09-b222-99dd64093045","Type":"ContainerStarted","Data":"cd7b9b9de02a9f8ce7c294b92625eca2acc5f449fbfe388520270c33605e5ab6"} Oct 30 21:53:59.450342 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.449737 2585 generic.go:356] "Generic (PLEG): container finished" podID="a689496985b50dbcdef398d32e6ac5d5" containerID="6b67fa8abd7eaeb196de8bd67004bd0bfd8d199508a685eb01288df84100a600" exitCode=0 Oct 30 21:53:59.450342 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.449819 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerDied","Data":"6b67fa8abd7eaeb196de8bd67004bd0bfd8d199508a685eb01288df84100a600"} Oct 30 21:53:59.458018 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.457497 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"c63e129e0cfb39daa6732f74d4a5cd8a","Type":"ContainerStarted","Data":"ea0e0880f065e1ff41d88f8bf4a082bb6c6770bf5c2e6d46ab8a269e800f0bc0"} Oct 30 21:53:59.570411 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:53:59.565738 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mv9k2" podStartSLOduration=2.927864745 podStartE2EDuration="7.565717513s" totalImagesPullingTime="4.637852768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:59.559551602 +0000 UTC m=+7.614058331" watchObservedRunningTime="2026-10-30 21:53:59.565717513 +0000 UTC m=+7.620224242" Oct 30 21:54:00.467818 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:00.467136 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4hdrj" event={"ID":"f56f8924-c137-477d-a859-6defeffe45d4","Type":"ContainerStarted","Data":"4797031e23051e0032dc70dc903f6ed1b9fbf21b8015a642b8dff4bdea619be4"} Oct 30 21:54:00.476691 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:00.476264 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerStarted","Data":"91d55c977701d617d86db87445cd65cafc0ae8c4f72d919e53c9e9e85a992501"} Oct 30 21:54:00.598594 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:00.598521 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" podStartSLOduration=8.598500665 podStartE2EDuration="8.598500665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:59.802809283 +0000 UTC m=+7.857316017" watchObservedRunningTime="2026-10-30 21:54:00.598500665 +0000 UTC m=+8.653007397" Oct 30 21:54:00.717310 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:00.717247 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4hdrj" podStartSLOduration=2.8552917410000003 podStartE2EDuration="8.717227236s" totalImagesPullingTime="5.861935495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:00.599788292 +0000 UTC m=+8.654295022" watchObservedRunningTime="2026-10-30 21:54:00.717227236 +0000 UTC m=+8.771733965" Oct 30 21:54:01.228147 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.228107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:01.228319 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.228208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:01.228384 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.228321 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:54:01.228443 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.228384 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:09.22836554 +0000 UTC m=+17.282872309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:54:01.228505 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.228457 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:54:01.228505 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.228496 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:54:09.228484391 +0000 UTC m=+17.282991117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.368663 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.368826 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.368906 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.368982 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.369047 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:01.370033 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.369183 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:01.429711 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.429677 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:01.429877 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.429822 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:54:01.429877 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.429836 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:54:01.429877 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.429845 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:54:01.429998 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:01.429886 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:54:09.429874874 +0000 UTC m=+17.484381593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:54:01.482698 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:01.482664 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" event={"ID":"b60aa182-3a90-40fd-b14d-620259e35523","Type":"ContainerStarted","Data":"cbdda1711dce93e52c6eac47fb59b523f57bb6dde896975e91f244c73d73fcb8"} Oct 30 21:54:02.922664 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:02.922630 2585 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:54:03.150686 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.150645 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:03.151210 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.151184 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:03.289902 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.289849 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" podStartSLOduration=11.289828743 podStartE2EDuration="11.289828743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:00.718443873 +0000 UTC m=+8.772950605" watchObservedRunningTime="2026-10-30 21:54:03.289828743 +0000 UTC m=+11.344335472" Oct 30 21:54:03.322472 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.322316 2585 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:54:02.922650851Z","UUID":"aff67ddb-6384-427a-97dc-a5d1f82533e2","Handler":null,"Name":"","Endpoint":""} Oct 30 21:54:03.323858 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.323841 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:54:03.323960 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.323870 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:54:03.368868 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.368843 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:03.368988 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.368866 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:03.368988 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:03.368951 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:03.369099 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.369009 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:03.369134 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:03.369096 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:03.369186 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:03.369171 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:03.487722 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.487688 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" event={"ID":"b60aa182-3a90-40fd-b14d-620259e35523","Type":"ContainerStarted","Data":"7cd87743df331b38f5cb62be9843fd930c96ba8ff04d21af005cb3e7293a9d97"} Oct 30 21:54:03.489141 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.489114 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x6pp9" event={"ID":"6da87eda-28f6-4a7c-a004-42dd0b6a0d58","Type":"ContainerStarted","Data":"b67224dc0c69b725e3d4f796288748989a348703dc8df5b8861ce3b759cc4914"} Oct 30 21:54:03.492119 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.492093 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="cc7dcb58162909a0efb241692aa0545da233152da9a0466e45f7f925d9db9e51" exitCode=0 Oct 30 21:54:03.492201 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.492156 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"cc7dcb58162909a0efb241692aa0545da233152da9a0466e45f7f925d9db9e51"} Oct 30 21:54:03.494765 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.494295 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rmh74" event={"ID":"b1b39b57-4d19-426b-982b-4e197e37736f","Type":"ContainerStarted","Data":"80ee9f99aa73760f25f7b479d3c824dc06aece255c363f90c182c51fff5e43ef"} Oct 30 21:54:03.606164 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.606070 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x6pp9" podStartSLOduration=3.189133049 podStartE2EDuration="11.606051389s" totalImagesPullingTime="8.41691834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:03.605335321 +0000 UTC m=+11.659842049" watchObservedRunningTime="2026-10-30 21:54:03.606051389 +0000 UTC m=+11.660558119" Oct 30 21:54:03.847540 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:03.847477 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rmh74" podStartSLOduration=3.279432298 podStartE2EDuration="11.847462703s" totalImagesPullingTime="8.568030405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:03.846688859 +0000 UTC m=+11.901195586" watchObservedRunningTime="2026-10-30 21:54:03.847462703 +0000 UTC m=+11.901969431" Oct 30 21:54:04.497677 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:04.497425 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcl4g" event={"ID":"5b6ed4c4-5043-4424-8b2d-826f478062c6","Type":"ContainerStarted","Data":"0797ba8e8a84e503622607980029b84cd52ed0760162b3ab475ab320493e88ca"} Oct 30 21:54:04.613442 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:04.613397 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fcl4g" podStartSLOduration=4.136652775 podStartE2EDuration="12.613384465s" totalImagesPullingTime="8.47673169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:04.613036092 +0000 UTC m=+12.667542831" watchObservedRunningTime="2026-10-30 21:54:04.613384465 +0000 UTC m=+12.667891192" Oct 30 21:54:05.368770 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.368740 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:05.368926 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.368789 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:05.368926 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.368885 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:05.369032 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:05.368919 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:05.369091 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:05.369036 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:05.369167 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:05.369145 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:05.501929 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.501892 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="b1e6f9440d362070c4067ff5c5b07d3fcf1da03499f66d3b36e868bf1cd3a9e2" exitCode=0 Oct 30 21:54:05.502524 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.501984 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"b1e6f9440d362070c4067ff5c5b07d3fcf1da03499f66d3b36e868bf1cd3a9e2"} Oct 30 21:54:05.505202 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.505158 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" event={"ID":"b60aa182-3a90-40fd-b14d-620259e35523","Type":"ContainerStarted","Data":"4e5f78e2e90ccb49d46483f131f290e5d1c0cba071881118b411799accebbe2f"} Oct 30 21:54:05.506879 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.506854 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9glb" event={"ID":"cf03a9d9-1567-4254-9747-d9836c323a2c","Type":"ContainerStarted","Data":"e6f616cc0b7f4199feb0296e14b21468752b1da6cee66d6235fb7c5017996790"} Oct 30 21:54:05.875194 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.875149 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w9glb" podStartSLOduration=3.0987079299999998 podStartE2EDuration="13.875135305s" totalImagesPullingTime="10.776427375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.745412387 +0000 UTC m=+13.799919116" watchObservedRunningTime="2026-10-30 21:54:05.875135305 +0000 UTC m=+13.929642024" Oct 30 21:54:05.875393 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:05.875374 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" podStartSLOduration=3.75971109 podStartE2EDuration="13.875367949s" totalImagesPullingTime="10.115656859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.87415199 +0000 UTC m=+13.928658718" watchObservedRunningTime="2026-10-30 21:54:05.875367949 +0000 UTC m=+13.929874677" Oct 30 21:54:07.368002 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:07.367973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:07.368002 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:07.367997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:07.368724 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:07.368094 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:07.368724 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:07.368154 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:07.368724 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:07.368149 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:07.368724 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:07.368203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:07.513382 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:07.513353 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="e5b10e195aa2f13d9d295b54df263a7a35d402601d7248176ccdc66aea883c29" exitCode=0 Oct 30 21:54:07.513565 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:07.513393 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"e5b10e195aa2f13d9d295b54df263a7a35d402601d7248176ccdc66aea883c29"} Oct 30 21:54:09.287120 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.286865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:09.287564 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.287145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:09.287564 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.287044 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:54:09.287564 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.287234 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:25.287215809 +0000 UTC m=+33.341722518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:54:09.287564 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.287274 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:54:09.287564 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.287325 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret podName:4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd nodeName:}" failed. No retries permitted until 2026-10-30 21:54:25.287310012 +0000 UTC m=+33.341816727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret") pod "global-pull-secret-syncer-zrjkd" (UID: "4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:54:09.367891 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.367867 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:09.368041 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.367894 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:09.368041 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.367991 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:09.368148 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.368051 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:09.368148 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.368109 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:09.368239 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.368170 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:09.488500 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:09.488447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:09.488667 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.488621 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:54:09.488667 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.488639 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:54:09.488667 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.488651 2585 projected.go:196] Error preparing data for projected volume kube-api-access-skpcs for pod openshift-network-diagnostics/network-check-target-g2jrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:54:09.488780 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:09.488700 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs podName:685b08d4-b70f-4693-bced-4fd0b99d952a nodeName:}" failed. No retries permitted until 2026-10-30 21:54:25.488687296 +0000 UTC m=+33.543194003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-skpcs" (UniqueName: "kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs") pod "network-check-target-g2jrf" (UID: "685b08d4-b70f-4693-bced-4fd0b99d952a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:54:10.522313 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.522249 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="ecc9a181c125c80e34741e17b1b6c080b5d3a1281fb8131ea737b92f85a100d6" exitCode=0 Oct 30 21:54:10.522815 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.522337 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"ecc9a181c125c80e34741e17b1b6c080b5d3a1281fb8131ea737b92f85a100d6"} Oct 30 21:54:10.525489 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525465 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"5a6915dc5c7691f4cd1dc55f0352623534990229a307cdf63c7c0b1850738a62"} Oct 30 21:54:10.525648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525521 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"a08069225ce7fec88c940d118c4cad831f0b4acb3aec552b26933918cacc17fc"} Oct 30 21:54:10.525648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525552 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"d9ec8e60c2fda7d24922e7b2b66cfb95c6f92a36b9d292adc6d98a6f3b377f9e"} Oct 30 21:54:10.525648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525604 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"e5d2afd85d36885cadbbedaa396fd93bc0d0215191d6425939b6ee8bb933ab7c"} Oct 30 21:54:10.525648 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525638 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"60dd219b2faf17980465bafcf921f5800247c742fcd8de6f48fe116aea7afe91"} Oct 30 21:54:10.525840 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:10.525667 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"0354b01d5165bc021b1525ee46406d018d9051d75deca4a1032b200c1506919a"} Oct 30 21:54:11.368851 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:11.368669 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:11.369026 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:11.368700 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:11.369026 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:11.368919 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:11.369026 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:11.368997 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:11.369026 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:11.368745 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:11.369163 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:11.369081 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:11.531072 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:11.531038 2585 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="c17e37f00175d4e9b9b38a72268491940f45ec0820089d28785e8321d99cc719" exitCode=0 Oct 30 21:54:11.531439 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:11.531078 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerDied","Data":"c17e37f00175d4e9b9b38a72268491940f45ec0820089d28785e8321d99cc719"} Oct 30 21:54:12.536222 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:12.536185 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" event={"ID":"cfc5a0a3-e851-4c62-8a13-0985bbb5068e","Type":"ContainerStarted","Data":"576809530cb7620892fff67e2623bfa42b3d4e99953bea8878eb392812635a86"} Oct 30 21:54:12.539109 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:12.539076 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"c7eb0dfe755f71af7f08f928d79882892043e0e87e1378de3ed8b31c34162d07"} Oct 30 21:54:12.659351 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:12.659268 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lgkpc" podStartSLOduration=7.537825467 podStartE2EDuration="20.659256241s" totalImagesPullingTime="13.121430774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.658939081 +0000 UTC m=+20.713445809" watchObservedRunningTime="2026-10-30 21:54:12.659256241 +0000 UTC m=+20.713762968" Oct 30 21:54:13.368777 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:13.368744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:13.368777 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:13.368771 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:13.368777 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:13.368783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:13.368981 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:13.368863 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:13.368981 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:13.368918 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:13.369048 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:13.369005 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:14.547273 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:14.547240 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" event={"ID":"86e6cf70-3f1a-40a1-b849-fcea928413c7","Type":"ContainerStarted","Data":"1717fd21a86c916949d815a33357901c15f225434b95203385fb435fc0060130"} Oct 30 21:54:14.547940 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:14.547516 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:14.561672 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:14.561645 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:14.782330 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:14.782235 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" podStartSLOduration=6.663512084 podStartE2EDuration="22.782218681s" totalImagesPullingTime="16.118706597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:14.732858582 +0000 UTC m=+22.787365309" watchObservedRunningTime="2026-10-30 21:54:14.782218681 +0000 UTC m=+22.836725411" Oct 30 21:54:15.367783 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.367750 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:15.367928 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.367815 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:15.367928 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.367843 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:15.367928 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:15.367913 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:15.368044 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:15.367966 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:15.368084 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:15.368059 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:15.548960 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.548939 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:54:15.549474 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.549341 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:15.563867 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:15.563849 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:16.535747 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:16.535713 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:16.536004 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:16.535821 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:54:16.536269 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:16.536250 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:16.550856 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:16.550835 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:54:17.243017 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.242794 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2jrf"] Oct 30 21:54:17.243156 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.243076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:17.243214 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:17.243176 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:17.243458 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.243430 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqpgq"] Oct 30 21:54:17.243560 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.243545 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:17.243673 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:17.243655 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:17.264490 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.264457 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrjkd"] Oct 30 21:54:17.264626 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.264559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:17.264686 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:17.264666 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:17.552360 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.552339 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:54:17.845365 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:17.845294 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:18.368133 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:18.368103 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:18.368284 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:18.368234 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:19.368397 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:19.368367 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:19.368797 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:19.368476 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:19.368797 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:19.368525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:19.368797 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:19.368630 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:20.368180 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:20.368152 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:20.368405 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:20.368266 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:21.368597 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:21.368520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:21.368920 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:21.368652 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqpgq" podUID="c10c2481-4327-49ef-a6b1-c922c6411099" Oct 30 21:54:21.368920 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:21.368725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:21.368920 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:21.368809 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2jrf" podUID="685b08d4-b70f-4693-bced-4fd0b99d952a" Oct 30 21:54:22.369944 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:22.369914 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:22.370481 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:22.370026 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrjkd" podUID="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" Oct 30 21:54:23.238120 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.238091 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeReady" Oct 30 21:54:23.238230 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.238205 2585 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:54:23.368241 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.368187 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:23.368361 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.368342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:23.425960 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.425936 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:54:23.426255 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.426235 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66njg\"" Oct 30 21:54:23.426255 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.426240 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:54:23.426318 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.426251 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:54:23.426318 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:23.426306 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fd64m\"" Oct 30 21:54:24.368441 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:24.368414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:24.424552 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:24.424532 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c7qtj\"" Oct 30 21:54:24.474202 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:24.474185 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:54:25.302722 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.302688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:25.302873 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.302727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:25.302873 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:25.302837 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:54:25.302943 ip-10-0-142-210 kubenswrapper[2585]: E1030 21:54:25.302906 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs podName:c10c2481-4327-49ef-a6b1-c922c6411099 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:57.302887695 +0000 UTC m=+65.357394407 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs") pod "network-metrics-daemon-bqpgq" (UID: "c10c2481-4327-49ef-a6b1-c922c6411099") : secret "metrics-daemon-secret" not found Oct 30 21:54:25.305999 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.305976 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:25.504166 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.504137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:25.506342 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.506321 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:25.570291 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.570243 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:25.602553 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:25.602536 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:26.070300 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:26.070258 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrjkd"] Oct 30 21:54:26.072886 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:26.072867 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2jrf"] Oct 30 21:54:26.074230 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:54:26.074209 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8442af_fc07_44d2_aeaf_b6ad5d77dcdd.slice/crio-b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d WatchSource:0}: Error finding container b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d: Status 404 returned error can't find the container with id b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d Oct 30 21:54:26.109668 ip-10-0-142-210 kubenswrapper[2585]: W1030 21:54:26.109646 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685b08d4_b70f_4693_bced_4fd0b99d952a.slice/crio-295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf WatchSource:0}: Error finding container 295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf: Status 404 returned error can't find the container with id 295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf Oct 30 21:54:26.570943 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:26.570911 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2jrf" event={"ID":"685b08d4-b70f-4693-bced-4fd0b99d952a","Type":"ContainerStarted","Data":"295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf"} Oct 30 21:54:26.572012 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:26.571993 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrjkd" event={"ID":"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd","Type":"ContainerStarted","Data":"b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d"} Oct 30 21:54:29.427955 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:29.427926 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x6pp9_6da87eda-28f6-4a7c-a004-42dd0b6a0d58/dns-node-resolver/0.log" Oct 30 21:54:29.870969 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:29.870932 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4hdrj_f56f8924-c137-477d-a859-6defeffe45d4/node-ca/0.log" Oct 30 21:54:30.582830 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:30.582785 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2jrf" event={"ID":"685b08d4-b70f-4693-bced-4fd0b99d952a","Type":"ContainerStarted","Data":"5ea6c0d07859622fc1cfd1a5146b2059bddb885c758743e63ce7fced76ec389b"} Oct 30 21:54:30.583277 ip-10-0-142-210 kubenswrapper[2585]: I1030 21:54:30.582939 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:31.960273 ip-10-0-142-210 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:54:31.980964 ip-10-0-142-210 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:54:31.981179 ip-10-0-142-210 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:54:31.981238 ip-10-0-142-210 systemd[1]: kubelet.service: Consumed 2.007s CPU time. Oct 30 21:54:32.003123 ip-10-0-142-210 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:54:32.073727 ip-10-0-142-210 kubenswrapper[4292]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:54:32.073727 ip-10-0-142-210 kubenswrapper[4292]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:54:32.073727 ip-10-0-142-210 kubenswrapper[4292]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:54:32.073727 ip-10-0-142-210 kubenswrapper[4292]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:54:32.075712 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075647 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.075712 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075712 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075716 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075720 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075724 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075727 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075730 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075733 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075735 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075738 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075741 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075744 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075746 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075749 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075751 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075764 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075767 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075771 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075775 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075778 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.075789 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075781 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075783 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075786 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075789 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075792 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075795 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075798 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075801 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075804 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075806 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075809 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075812 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075814 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075817 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075819 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075822 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075825 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075827 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075830 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075832 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.076258 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075835 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075838 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075840 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075842 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075845 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075848 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075850 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075853 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075856 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075858 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075861 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075863 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075866 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075869 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075871 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075874 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075877 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075879 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075882 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075884 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.076769 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075887 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075891 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075896 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075899 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075901 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075904 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075907 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075909 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075912 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075914 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075917 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075919 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075922 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075925 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075927 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075930 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075932 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075935 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075937 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075940 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.077268 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075949 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075952 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075955 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075957 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075960 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075963 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075965 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075969 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075972 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075975 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075977 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075980 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075983 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075986 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075988 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075991 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075993 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075996 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.075998 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.077811 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076001 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076003 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076006 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076008 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076011 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076013 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076017 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076863 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076871 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076875 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076879 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076882 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076885 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076888 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076898 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076901 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076903 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076907 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076910 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076913 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.078280 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076916 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076919 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076922 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076924 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076927 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076930 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076933 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076935 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076938 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076940 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076943 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076946 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076948 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076951 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076954 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076956 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076959 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076961 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076964 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.078775 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076966 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076968 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076971 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076974 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076987 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076991 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076995 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.076998 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077006 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077009 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077012 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077014 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077017 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077020 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077023 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077025 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077028 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077030 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077033 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077035 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.079270 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077038 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077041 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077043 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077046 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077048 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077051 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077053 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077056 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077058 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077062 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077065 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077068 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077071 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077073 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077076 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077078 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077081 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077084 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077086 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.079766 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077089 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077091 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077099 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077102 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077105 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077107 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077110 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077112 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077114 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077117 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077120 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077122 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077125 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077128 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077133 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077137 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077141 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077144 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077147 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.080262 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077149 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077152 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077155 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077158 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077161 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077163 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077166 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077169 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077171 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077174 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077176 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077179 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077182 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077184 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077187 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077189 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077734 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077740 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077743 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.080749 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077745 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077748 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077751 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077753 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077756 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077758 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077761 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077763 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077766 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077769 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077771 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077774 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077776 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077779 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077782 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077786 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077788 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077791 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077794 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077796 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.081202 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077799 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077802 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077805 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077807 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077810 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077813 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077815 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077820 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077823 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077830 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077833 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077836 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077838 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077841 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077844 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077847 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077849 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077852 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077855 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.081742 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077857 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077860 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077862 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077865 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077868 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077870 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077873 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077876 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077878 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077881 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077884 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077887 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077889 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077892 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077894 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077897 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077899 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077902 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077906 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.082204 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077910 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077913 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077916 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077918 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077927 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077930 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077932 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077935 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077937 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077940 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077943 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077945 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077950 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077953 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077956 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077958 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077961 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077963 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077966 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.082687 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077968 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077971 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077973 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077976 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077978 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077981 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077983 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077986 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077988 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077990 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077993 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077996 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.077998 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078001 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078003 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078006 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078008 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078010 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078019 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078022 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.083456 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078025 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078027 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078030 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078033 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078036 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078041 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078128 4292 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078135 4292 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078147 4292 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078151 4292 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078156 4292 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078159 4292 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078164 4292 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078168 4292 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078171 4292 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078174 4292 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078178 4292 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078181 4292 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078184 4292 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078187 4292 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078190 4292 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078193 4292 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078196 4292 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078199 4292 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:54:32.084190 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078206 4292 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078209 4292 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078212 4292 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078215 4292 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078218 4292 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078222 4292 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078225 4292 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078235 4292 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078240 4292 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078243 4292 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078247 4292 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078250 4292 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078253 4292 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078257 4292 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078262 4292 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078265 4292 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078268 4292 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078271 4292 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078274 4292 flags.go:64] FLAG: --enable-server="true" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078277 4292 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078285 4292 flags.go:64] FLAG: --event-burst="100" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078289 4292 flags.go:64] FLAG: --event-qps="50" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078292 4292 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078295 4292 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:54:32.084801 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078298 4292 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078302 4292 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078305 4292 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078308 4292 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078311 4292 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078314 4292 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078317 4292 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078320 4292 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078323 4292 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078326 4292 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078329 4292 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078332 4292 flags.go:64] FLAG: --feature-gates="" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078339 4292 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078342 4292 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078345 4292 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078349 4292 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078359 4292 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078362 4292 flags.go:64] FLAG: --help="false" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078365 4292 flags.go:64] FLAG: --hostname-override="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078369 4292 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078372 4292 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078382 4292 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078386 4292 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:54:32.085408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078390 4292 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078393 4292 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078396 4292 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078399 4292 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078402 4292 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078405 4292 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078409 4292 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078412 4292 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078415 4292 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078417 4292 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078421 4292 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078424 4292 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078427 4292 flags.go:64] FLAG: --lock-file="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078430 4292 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078433 4292 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078435 4292 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078441 4292 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078444 4292 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078447 4292 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078450 4292 flags.go:64] FLAG: --logging-format="text" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078453 4292 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078462 4292 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078465 4292 flags.go:64] FLAG: --manifest-url="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078468 4292 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078473 4292 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:54:32.085997 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078477 4292 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078487 4292 flags.go:64] FLAG: --max-pods="110" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078490 4292 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078493 4292 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078496 4292 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078501 4292 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078504 4292 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078507 4292 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078510 4292 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078517 4292 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078520 4292 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078523 4292 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078526 4292 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078529 4292 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078532 4292 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078535 4292 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078538 4292 flags.go:64] FLAG: --port="10250" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078541 4292 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078544 4292 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02e6328d3198bef35" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078547 4292 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078550 4292 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078553 4292 flags.go:64] FLAG: --register-node="true" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078556 4292 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078560 4292 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078562 4292 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078565 4292 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:54:32.086594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078568 4292 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078589 4292 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078594 4292 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078599 4292 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078603 4292 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078608 4292 flags.go:64] FLAG: --runonce="false" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078613 4292 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078617 4292 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078626 4292 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078629 4292 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078632 4292 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078637 4292 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078640 4292 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078644 4292 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078646 4292 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078649 4292 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078652 4292 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078655 4292 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078658 4292 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078661 4292 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078664 4292 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078669 4292 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078672 4292 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078675 4292 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078682 4292 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:54:32.087329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078685 4292 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078688 4292 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078691 4292 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078694 4292 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078697 4292 flags.go:64] FLAG: --v="2" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078701 4292 flags.go:64] FLAG: --version="false" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078705 4292 flags.go:64] FLAG: --vmodule="" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078709 4292 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.078712 4292 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078893 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078897 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078900 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078903 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078907 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078910 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078912 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078920 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078924 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078928 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078931 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078933 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078936 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.088033 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078939 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078941 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078944 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078947 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078949 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078952 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078955 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078957 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078960 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078963 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078966 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078969 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078979 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078982 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078985 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078987 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078990 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078994 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.078998 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079001 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.088637 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079004 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079007 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079010 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079014 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079017 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079019 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079022 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079031 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079035 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079038 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079041 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079044 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079046 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079049 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079051 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079054 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079057 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079059 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079065 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.089160 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079068 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079071 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079073 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079076 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079078 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079081 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079083 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079086 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079089 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079091 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079094 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079096 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079099 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079102 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079104 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079107 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079111 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079113 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079116 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.089709 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079119 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079121 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079130 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079133 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079136 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079139 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079142 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079144 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079147 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079149 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079152 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079154 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079157 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079160 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079162 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079164 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079167 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079169 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079172 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.090341 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079175 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079179 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079182 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079185 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079187 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079190 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079192 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079195 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079198 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079200 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079204 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079207 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079210 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079212 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079215 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.090835 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079219 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.079251 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079296 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079301 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079304 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079307 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079310 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079313 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079316 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079319 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079321 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079324 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079327 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079329 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079333 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079337 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.091264 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079341 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079344 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079346 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079349 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079352 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079354 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079357 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079359 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079362 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079364 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079367 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079369 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079372 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079375 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079378 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079381 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079385 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079388 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079396 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079398 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.092032 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079401 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079404 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079406 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079409 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079411 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079414 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079417 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079419 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079422 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079424 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079426 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079429 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079432 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079434 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079437 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079439 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079442 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079445 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079447 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079450 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.092524 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079452 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079455 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079458 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079462 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079466 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079469 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079471 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079474 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079478 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079481 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079483 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079491 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079494 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079497 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079500 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079503 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079505 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079508 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079510 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.093073 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079513 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079515 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079518 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079520 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079528 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079531 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079534 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079536 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079539 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079541 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079544 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079546 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079549 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079552 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079555 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079557 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079560 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079563 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079565 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.093587 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079568 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079585 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079591 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079595 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079599 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079603 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079612 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079615 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079618 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079621 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079625 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079627 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079630 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.079632 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.083915 4292 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.083936 4292 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.083959 4292 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.083967 4292 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084566 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.094109 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084600 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084604 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084607 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084610 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084613 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084616 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084619 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084623 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084631 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084634 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084637 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084639 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084642 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084645 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084647 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084650 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084653 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084655 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084658 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.094777 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084661 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084663 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084666 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084669 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084672 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084675 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084678 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084681 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084683 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084687 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084689 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084692 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084695 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084698 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084701 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084704 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084706 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084709 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084711 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084714 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.095420 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084716 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084719 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084721 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084724 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084727 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084729 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084732 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084734 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084737 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084740 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084742 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084745 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084747 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084749 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084752 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084755 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084757 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084760 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084762 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084765 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.096009 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084768 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084771 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084774 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084776 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084779 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084782 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084784 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084786 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084789 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084792 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084794 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084796 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084799 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084801 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084804 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084807 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084809 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084812 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084815 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084818 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.096484 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084821 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084824 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084826 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084829 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084831 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084834 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084836 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084839 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084842 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084844 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084847 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084849 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084852 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084854 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084857 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084859 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084862 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084865 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084867 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.096974 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084870 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084874 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084878 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084881 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084884 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084887 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.084890 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.084919 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085053 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085058 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085062 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085065 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085068 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085076 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085079 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085082 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.097465 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085085 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085088 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085091 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085093 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085096 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085098 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085101 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085104 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085106 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085109 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085111 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085114 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085117 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085119 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085122 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085125 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085127 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085130 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085133 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.097874 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085135 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085138 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085140 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085143 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085145 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085148 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085151 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085154 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085157 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085160 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085164 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085172 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085176 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085178 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085181 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085184 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085187 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085189 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085192 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.098323 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085195 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085197 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085200 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085202 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085206 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085208 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085211 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085213 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085217 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085221 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085224 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085227 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085230 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085233 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085236 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085238 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085241 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085244 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085246 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.098785 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085249 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085252 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085254 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085257 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085261 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085263 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085271 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085274 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085277 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085280 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085283 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085285 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085288 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085291 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085293 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085296 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085298 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085301 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085304 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.099263 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085306 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085309 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085312 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085314 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085317 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085319 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085322 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085325 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085329 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085332 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085335 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085338 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085340 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085343 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085346 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085348 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085351 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085354 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085357 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085360 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.099868 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085369 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085372 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.085397 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085432 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085436 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085440 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085443 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085446 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085449 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085452 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085455 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085457 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085460 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085462 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085465 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085467 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:54:32.100666 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085470 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085473 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085475 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085478 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085480 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085483 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085487 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085491 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085494 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085497 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085500 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085502 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085505 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085508 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085512 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085515 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085518 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085525 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085528 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085531 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:54:32.101305 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085534 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085536 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085539 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085541 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085544 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085546 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085549 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085551 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085554 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085557 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085560 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085562 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085565 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085567 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085570 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085584 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085587 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085590 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085592 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085595 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:54:32.102183 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085597 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085600 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085603 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085606 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085608 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085611 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085614 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085616 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085619 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085622 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085630 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085633 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085635 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085638 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085641 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085644 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085646 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085649 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085651 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:54:32.102801 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085654 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085657 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085660 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085662 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085664 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085667 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085670 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085673 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085675 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085678 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085680 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085683 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085686 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085688 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085691 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085695 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085698 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085702 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085705 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:54:32.103286 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085708 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085711 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085713 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085716 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085718 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085726 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085729 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085732 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085735 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085737 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085740 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085742 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085745 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085748 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:32.085750 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.085800 4292 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.086745 4292 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.086788 4292 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:54:32.103806 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.086991 4292 server.go:1014] "Starting client certificate rotation" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.087141 4292 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.087198 4292 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:48:52 +0000 UTC" deadline="2028-07-22 19:35:59.068580826 +0000 UTC" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.087253 4292 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15141h41m26.981330978s" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.087429 4292 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.088472 4292 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.089787 4292 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.091289 4292 log.go:25] "Validated CRI v1 image API" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.092061 4292 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:54:32.104269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.093340 4292 fs.go:135] Filesystem UUIDs: map[44bdeef8-8e21-4464-b53a-f73ca282ede9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e1804596-ffae-45b7-b6f1-88554366a9d5:/dev/nvme0n1p3] Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.093358 4292 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b6ed4c4-5043-4424-8b2d-826f478062c6/volumes/kubernetes.io~projected/kube-api-access-nxx44:{mountpoint:/var/lib/kubelet/pods/5b6ed4c4-5043-4424-8b2d-826f478062c6/volumes/kubernetes.io~projected/kube-api-access-nxx44 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/685b08d4-b70f-4693-bced-4fd0b99d952a/volumes/kubernetes.io~projected/kube-api-access-skpcs:{mountpoint:/var/lib/kubelet/pods/685b08d4-b70f-4693-bced-4fd0b99d952a/volumes/kubernetes.io~projected/kube-api-access-skpcs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6da87eda-28f6-4a7c-a004-42dd0b6a0d58/volumes/kubernetes.io~projected/kube-api-access-w7tv8:{mountpoint:/var/lib/kubelet/pods/6da87eda-28f6-4a7c-a004-42dd0b6a0d58/volumes/kubernetes.io~projected/kube-api-access-w7tv8 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~projected/kube-api-access-2whbs:{mountpoint:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~projected/kube-api-access-2whbs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81c0006-0d2d-4e09-b222-99dd64093045/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a81c0006-0d2d-4e09-b222-99dd64093045/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1b39b57-4d19-426b-982b-4e197e37736f/volumes/kubernetes.io~projected/kube-api-access-z8xrf:{mountpoint:/var/lib/kubelet/pods/b1b39b57-4d19-426b-982b-4e197e37736f/volumes/kubernetes.io~projected/kube-api-access-z8xrf major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60aa182-3a90-40fd-b14d-620259e35523/volumes/kubernetes.io~projected/kube-api-access-bw8d8:{mountpoint:/var/lib/kubelet/pods/b60aa182-3a90-40fd-b14d-620259e35523/volumes/kubernetes.io~projected/kube-api-access-bw8d8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c10c2481-4327-49ef-a6b1-c922c6411099/volumes/kubernetes.io~projected/kube-api-access-dg9ww:{mountpoint:/var/lib/kubelet/pods/c10c2481-4327-49ef-a6b1-c922c6411099/volumes/kubernetes.io~projected/kube-api-access-dg9ww major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~projected/kube-api-access-8426f:{mountpoint:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~projected/kube-api-access-8426f major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc5a0a3-e851-4c62-8a13-0985bbb5068e/volumes/kubernetes.io~projected/kube-api-access-8tw6w:{mountpoint:/var/lib/kubelet/pods/cfc5a0a3-e851-4c62-8a13-0985bbb5068e/volumes/kubernetes.io~projected/kube-api-access-8tw6w major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f56f8924-c137-477d-a859-6defeffe45d4/volumes/kubernetes.io~projected/kube-api-access-47wl4:{mountpoint:/var/lib/kubelet/pods/f56f8924-c137-477d-a859-6defeffe45d4/volumes/kubernetes.io~projected/kube-api-access-47wl4 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a9e368295cf25531e86155cb66f87152e8af1fd87971f0f29255ce415c7c6f5b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/384fb13117b4082be481aed05bd80f7a53d9a4b761d66599fc5ce032e8982da7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/536ce009bbfcd5b361eaab4cdce4ee16f4c464759d5edec15810723ac99698fb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e189ba072b897b6de6fc7481628e16652d815eb85f713b79c740c9f470a37c11/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/299e102db9632796b81fe53985c0288f88f81be040ef3aaf2d980a87a9bd2379/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/80303552d6b3af931d23cfc8cbd23a0d86854064f8ddc93822b6ae12929f204c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/53bb4c85424e9248bf861dfa66a289396ebc10528b991248338a53f7be0dbe26/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b247c5fe52a62ca81767b24d2935b43155dfd1ae771351b40e2b5760d346e4c4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/05f5c94cce2bd986eb94f1f9888fa2bfb636fd58e9254f944ac743ffdc7e92e1/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/aa5e59637b4a773adc9dda3af82784f71cbd0b8f54d6d2debd393f52dc92aa0c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/b3a3b478ede8e6f31ab79c4201216b4f018f002f77d94557c7330716058e9551/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2e329fc2e117464a44d075d8cf71157419fc44003f3ddcb61a3546ba97d1836e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c9d03c6eb1579fc16c19c5f621ce55c982f9496f2c29895a1575c51343479d89/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8cee11b5665e17f915b45392f99fb2c467c6b5b2e764b62f4a0ed9c4b29097e9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/c41c130e86269f4ecc49f0b7c6bfa4c5a890f809d39be46362b6cdc808c6c515/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/b1735cff60006400150a5c99bdb2cae37e76b205a5abad78970a1fc660e59fff/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c954dc0e529b1ee26473623f9b558211e9e710898893920a24e35227556055fe/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4302617a6fc8367bbcf39216f7a3e2e73baa8bd7794bd0bd9229d2984b6f9760/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/80e06e7f5f631c764af58d7afcaa764f7b7afd30ff15216db9c1e12e8f68612a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0741ac7d5a3bfe928a2ef92e0c14bb6bcb54009fae48fba1ab8712f59cb80731/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/51bfbc8e02013f92769b9bd6bf310e1313b34cea516dba9d5ee37d266c1efdb4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/8ef7e0f3f9abe234b31b36307f928f85df95bca455560775659da80cf2c3dabc/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/6e82b8985198566e59b7000c443efd9b9aaeea4ff0b5840a19890ff2daa68180/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/91dd6e9d494c375edfcaff385491b281150b44bea3bbbd94d40423e4fb0146a7/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ae2aa6677c30e3a0a5ad98f4a2d3b6f85dc9870fbbdd916284705087deb89e34/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4fb47a49fd9f72a6f1ffcaeec55a19a843eb3f9a000ac12f9ac540c2742b26cc/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b025994f511b9bfe7a8f286559a42ac78dc087f680a041354c47829404718415/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d0616a8b7defcac7902105ef2c465deab3ebb77d7849331452fb4e2d789fc997/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/876aed826c41dd21ec4cfdbae357f5c279ce438e36f7d9254341b1792b68fe4d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1f21d43fd8cc2780dfaf1dfb25fcfb9e0b353b3db3488a8da630c864008cf102/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2f0d8a5d1b8bfe542295e2f4ee8052ee5e3fe2ae4e1221b23d6351601d53ecd1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/be172f64dad62ef0ad4c1be953af5fc5da3f4151dabd739f916ddde58e96769b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e149029ef84fc527f6569d9780bf7817a9f04a48a253b9bd5296149594f2d3e4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/556e985ee55818c4152015660e53ba9b2293b1e54b90fdcb10dc0fe9585d6516/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/602e15d00de6bd6af22686d5b46599c76d3a7830acb81e26f90da7f31b323faf/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.098601 4292 manager.go:217] Machine: {Timestamp:2026-10-30 21:54:32.09815737 +0000 UTC m=+0.066976925 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099739 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2318c1ed0e34134f87fe622d6f94f2 SystemUUID:ec2318c1-ed0e-3413-4f87-fe622d6f94f2 BootID:a057bbe0-bccf-4e56-a218-46fd97492341 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cc9a0e0979f47b11837632c9003bad5df960f2f9d998c57806c583331aefa00/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a81c0006-0d2d-4e09-b222-99dd64093045/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b60aa182-3a90-40fd-b14d-620259e35523/volumes/kubernetes.io~projected/kube-api-access-bw8d8 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f56c80692af5cd17579b321fd2407d615909fcb3cfd0b7631f5e4baeb11a837/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6da87eda-28f6-4a7c-a004-42dd0b6a0d58/volumes/kubernetes.io~projected/kube-api-access-w7tv8 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1b39b57-4d19-426b-982b-4e197e37736f/volumes/kubernetes.io~projected/kube-api-access-z8xrf DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86e6cf70-3f1a-40a1-b849-fcea928413c7/volumes/kubernetes.io~projected/kube-api-access-2whbs DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b05e7ef5bfbe8fecd24aa0a91bf1a0371f47002ac140adb55270b149aa933b6b/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/295d82679593091d5c8c828f2ca4d7383ad8e20e4a9a5506f489026ea09bc4bf/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c10c2481-4327-49ef-a6b1-c922c6411099/volumes/kubernetes.io~projected/kube-api-access-dg9ww DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65afafa9d9ad04640d20afab252dac14e7aff0bd43fb3c67a966a1be134c5929/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6a21084a6f51f94c9779cd20389722f760ff9306eca5ef3f9a2f62c664ce76d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b6ed4c4-5043-4424-8b2d-826f478062c6/volumes/kubernetes.io~projected/kube-api-access-nxx44 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f56f8924-c137-477d-a859-6defeffe45d4/volumes/kubernetes.io~projected/kube-api-access-47wl4 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cfb88073e8ba93a2397753e7557adb2aab291c256b8024bdf1174199eba9b8a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a82e3d73d9b14e763cb915bb4faa166ee9caf16271e0a059422bc6df71bd2991/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc5a0a3-e851-4c62-8a13-0985bbb5068e/volumes/kubernetes.io~projected/kube-api-access-8tw6w DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf03a9d9-1567-4254-9747-d9836c323a2c/volumes/kubernetes.io~projected/kube-api-access-8426f DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/414ff6091a54eb9568eb89f28e915e637c7f0a31621ba9147f31593c3544232c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/685b08d4-b70f-4693-bced-4fd0b99d952a/volumes/kubernetes.io~projected/kube-api-access-skpcs DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a5499d10f570a6d264d918a631104e4dd690e028b894a80725a71d6a99eca57/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:295d82679593091 MacAddress:ea:8a:b7:e3:1e:0b Speed:10000 Mtu:8901} {Name:b6a21084a6f51f9 MacAddress:8a:a5:87:41:5d:de Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:75:4b:ba:f7:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:64:96:54:a5:e8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:75:4b:ba:f7:c5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:7e:4d:12:76:72 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:9d:34:1e:4a:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.098808 4292 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.098844 4292 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099104 4292 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099124 4292 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-210.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099240 4292 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099248 4292 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099263 4292 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099281 4292 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099348 4292 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099492 4292 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099606 4292 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099641 4292 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:54:32.104517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099650 4292 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099660 4292 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099668 4292 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.099682 4292 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.103087 4292 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.103107 4292 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:54:32.105084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.103569 4292 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:54:32.105434 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105420 4292 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:54:32.105434 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105433 4292 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:54:32.105544 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105534 4292 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:54:32.105745 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105733 4292 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:54:32.105784 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105764 4292 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:54:32.105859 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105849 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105864 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105871 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105877 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105882 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105888 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105893 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:54:32.105901 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105899 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105905 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105911 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105919 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105928 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105955 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:54:32.106229 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.105961 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:54:32.109043 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.109031 4292 server.go:1263] "Started kubelet" Oct 30 21:54:32.109657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.109098 4292 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:54:32.109657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.109148 4292 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:54:32.109657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.109235 4292 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:54:32.109657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.109648 4292 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:54:32.109531 ip-10-0-142-210 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:54:32.110480 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.110343 4292 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:54:32.111916 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.111898 4292 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:54:32.112042 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.111927 4292 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:54:32.112150 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.112127 4292 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:48:53 +0000 UTC" deadline="2028-04-21 10:51:14.544246633 +0000 UTC" Oct 30 21:54:32.112207 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.112151 4292 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12924h56m42.43209941s" Oct 30 21:54:32.112266 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.112208 4292 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:54:32.112266 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.112217 4292 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:54:32.112348 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.112317 4292 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:54:32.112448 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.112391 4292 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" Oct 30 21:54:32.113596 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113528 4292 factory.go:153] Registering CRI-O factory Oct 30 21:54:32.113596 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113548 4292 factory.go:223] Registration of the crio container factory successfully Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113568 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd" volumeName="kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113644 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6ed4c4-5043-4424-8b2d-826f478062c6" volumeName="kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113665 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da87eda-28f6-4a7c-a004-42dd0b6a0d58" volumeName="kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113679 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e6cf70-3f1a-40a1-b849-fcea928413c7" volumeName="kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113691 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e6cf70-3f1a-40a1-b849-fcea928413c7" volumeName="kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113703 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1b39b57-4d19-426b-982b-4e197e37736f" volumeName="kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113715 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1b39b57-4d19-426b-982b-4e197e37736f" volumeName="kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113727 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf03a9d9-1567-4254-9747-d9836c323a2c" volumeName="kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113742 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="685b08d4-b70f-4693-bced-4fd0b99d952a" volumeName="kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113763 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da87eda-28f6-4a7c-a004-42dd0b6a0d58" volumeName="kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113775 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e6cf70-3f1a-40a1-b849-fcea928413c7" volumeName="kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113788 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81c0006-0d2d-4e09-b222-99dd64093045" volumeName="kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca" seLinuxMountContext="" Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113667 4292 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113869 4292 factory.go:55] Registering systemd factory Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113879 4292 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113895 4292 factory.go:103] Registering Raw factory Oct 30 21:54:32.113754 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.113918 4292 manager.go:1199] Started watching for new ooms in manager Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114340 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81c0006-0d2d-4e09-b222-99dd64093045" volumeName="kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114357 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10c2481-4327-49ef-a6b1-c922c6411099" volumeName="kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114375 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf03a9d9-1567-4254-9747-d9836c323a2c" volumeName="kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114388 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" volumeName="kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114400 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e6cf70-3f1a-40a1-b849-fcea928413c7" volumeName="kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114411 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1b39b57-4d19-426b-982b-4e197e37736f" volumeName="kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114424 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60aa182-3a90-40fd-b14d-620259e35523" volumeName="kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114435 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" volumeName="kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114449 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" volumeName="kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114461 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f56f8924-c137-477d-a859-6defeffe45d4" volumeName="kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114476 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6ed4c4-5043-4424-8b2d-826f478062c6" volumeName="kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114488 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e6cf70-3f1a-40a1-b849-fcea928413c7" volumeName="kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114500 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf03a9d9-1567-4254-9747-d9836c323a2c" volumeName="kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114512 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" volumeName="kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114527 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f56f8924-c137-477d-a859-6defeffe45d4" volumeName="kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca" seLinuxMountContext="" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114540 4292 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:54:32.115181 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114546 4292 manager.go:319] Starting recovery of all containers Oct 30 21:54:32.117373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.114548 4292 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:54:32.137414 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.137356 4292 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:54:32.138665 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.138641 4292 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:54:32.138665 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.138663 4292 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:54:32.138817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.138679 4292 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:54:32.138918 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.138897 4292 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:54:32.145261 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145238 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="c17e37f00175d4e9b9b38a72268491940f45ec0820089d28785e8321d99cc719" exitCode=0 Oct 30 21:54:32.145261 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145257 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="ecc9a181c125c80e34741e17b1b6c080b5d3a1281fb8131ea737b92f85a100d6" exitCode=0 Oct 30 21:54:32.145374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145266 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="e5b10e195aa2f13d9d295b54df263a7a35d402601d7248176ccdc66aea883c29" exitCode=0 Oct 30 21:54:32.145374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145273 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="b1e6f9440d362070c4067ff5c5b07d3fcf1da03499f66d3b36e868bf1cd3a9e2" exitCode=0 Oct 30 21:54:32.145374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145278 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="cc7dcb58162909a0efb241692aa0545da233152da9a0466e45f7f925d9db9e51" exitCode=0 Oct 30 21:54:32.145374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.145284 4292 generic.go:356] "Generic (PLEG): container finished" podID="cfc5a0a3-e851-4c62-8a13-0985bbb5068e" containerID="63e6b63128b3474c535e2442dbd3a5e30591494e70eaeaa65bcf07c108b38605" exitCode=0 Oct 30 21:54:32.160418 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.160389 4292 generic.go:356] "Generic (PLEG): container finished" podID="a689496985b50dbcdef398d32e6ac5d5" containerID="6b67fa8abd7eaeb196de8bd67004bd0bfd8d199508a685eb01288df84100a600" exitCode=0 Oct 30 21:54:32.173873 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.173857 4292 manager.go:324] Recovery completed Oct 30 21:54:32.184657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.184639 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.185640 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.185620 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.185705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.185653 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.185705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.185665 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.189201 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189182 4292 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:54:32.189201 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189197 4292 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:54:32.189293 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189202 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.189293 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189212 4292 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:54:32.189357 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189325 4292 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:54:32.189357 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189341 4292 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:54:32.189416 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189362 4292 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:54:32.189416 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189368 4292 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:54:32.189416 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189373 4292 policy_none.go:50] "Start" Oct 30 21:54:32.189416 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189381 4292 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:54:32.189416 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189390 4292 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:54:32.189993 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.189977 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.190051 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.190004 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.190051 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.190015 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.200276 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.200251 4292 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:54:32.200276 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.200279 4292 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:54:32.200276 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.200285 4292 policy_none.go:44] "Start" Oct 30 21:54:32.203920 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.203898 4292 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:54:32.207149 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207134 4292 manager.go:341] "Starting Device Plugin manager" Oct 30 21:54:32.207223 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.207156 4292 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:54:32.207223 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207168 4292 server.go:87] "Starting device plugin registration server" Oct 30 21:54:32.207468 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207452 4292 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:54:32.207529 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207468 4292 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:54:32.207621 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207603 4292 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:54:32.207717 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207689 4292 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:54:32.207789 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207722 4292 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:54:32.207789 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207728 4292 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:54:32.207882 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.207796 4292 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:54:32.207712803Z","UUID":"e8abffd6-1810-4796-aebb-1e6a110ebb13","Handler":null,"Name":"","Endpoint":""} Oct 30 21:54:32.208676 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.208656 4292 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:54:32.208764 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.208692 4292 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" Oct 30 21:54:32.209215 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.209198 4292 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:54:32.209307 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.209221 4292 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:54:32.239791 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.239306 4292 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal"] Oct 30 21:54:32.239858 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.239833 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.243567 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.243550 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.243643 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.243588 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.243643 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.243599 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.243800 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.243787 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.243920 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.243906 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.244556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244534 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.244556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244553 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.244715 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244566 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.244715 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244589 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.244715 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244598 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.244715 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244607 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.244715 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.244704 4292 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.244953 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244882 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerStarted","Data":"91d55c977701d617d86db87445cd65cafc0ae8c4f72d919e53c9e9e85a992501"} Oct 30 21:54:32.244953 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244924 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.244953 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerDied","Data":"6b67fa8abd7eaeb196de8bd67004bd0bfd8d199508a685eb01288df84100a600"} Oct 30 21:54:32.244953 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244946 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"a689496985b50dbcdef398d32e6ac5d5","Type":"ContainerStarted","Data":"80954c884d4f58ad307b8fd0f21c4e620ae7b05839dcfe8cdf28533b501a1eaa"} Oct 30 21:54:32.245094 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244963 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"c63e129e0cfb39daa6732f74d4a5cd8a","Type":"ContainerStarted","Data":"ea0e0880f065e1ff41d88f8bf4a082bb6c6770bf5c2e6d46ab8a269e800f0bc0"} Oct 30 21:54:32.245094 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.244982 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" event={"ID":"c63e129e0cfb39daa6732f74d4a5cd8a","Type":"ContainerStarted","Data":"339e2d706092e511759b945e61caeedf6d25c860cebed89bce3f45d1b65be624"} Oct 30 21:54:32.245705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.245688 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.245757 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.245715 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.245757 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.245724 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.245853 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.245842 4292 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-210.us-west-1.compute.internal\" not found" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.308183 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.308164 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:54:32.308989 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.308974 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:54:32.309061 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.308998 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:54:32.309061 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.309008 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:54:32.309061 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.309024 4292 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.324979 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.324959 4292 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:54:32.325085 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.325073 4292 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:54:32.325125 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.325102 4292 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:54:32.368030 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.368010 4292 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:54:32.375840 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.375820 4292 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:54:32.413875 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.413856 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.425887 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.425866 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.425943 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.425889 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.425943 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.425907 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.464428 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.464411 4292 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.464493 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.464460 4292 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.464917 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.464899 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:54:32.464979 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.464934 4292 setters.go:546] "Node became not ready" node="ip-10-0-142-210.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:54:32Z","lastTransitionTime":"2026-10-30T21:54:32Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:54:32.472524 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.472504 4292 nodeinfomanager.go:471] existing CSINode "ip-10-0-142-210.us-west-1.compute.internal" is owned by different node (oldNodeID="a4946311-3dec-4fb3-857e-bc2786f0c074", newNodeID=""), cleaning up... Oct 30 21:54:32.477297 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.477281 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:54:32.477346 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.477317 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.477346 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.477328 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526154 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526103 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526154 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526130 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526154 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526314 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526193 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526314 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526193 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c63e129e0cfb39daa6732f74d4a5cd8a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"c63e129e0cfb39daa6732f74d4a5cd8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.526314 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.526238 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a689496985b50dbcdef398d32e6ac5d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" (UID: \"a689496985b50dbcdef398d32e6ac5d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.541293 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.541270 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:54:32.541364 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:32.541313 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:32.744788 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.744767 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-210.us-west-1.compute.internal" event="NodeReady" Oct 30 21:54:32.744885 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:32.744825 4292 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:54:33.100682 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.100656 4292 apiserver.go:52] "Watching apiserver" Oct 30 21:54:33.169662 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.169639 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:33.169772 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.169742 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:33.210761 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.210741 4292 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:54:33.211103 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.211085 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-fcl4g","openshift-ovn-kubernetes/ovnkube-node-4nsbz","kube-system/global-pull-secret-syncer-zrjkd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6","openshift-cluster-node-tuning-operator/tuned-w9glb","openshift-dns/node-resolver-x6pp9","openshift-image-registry/node-ca-4hdrj","openshift-multus/multus-rmh74","kube-system/konnectivity-agent-mv9k2","kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-lgkpc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-bqpgq","openshift-network-diagnostics/network-check-target-g2jrf"] Oct 30 21:54:33.214649 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.214633 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.228968 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.228950 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229044 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.228975 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229044 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.228996 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.229044 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229022 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229134 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229080 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229163 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229127 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229191 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229168 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229222 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229201 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx44\" (UniqueName: \"kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.229251 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229220 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229251 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229351 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229339 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229389 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229375 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229418 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229407 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229453 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229431 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.229490 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229455 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8d8\" (UniqueName: \"kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.229490 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229481 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229561 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229561 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229534 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.229673 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229602 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.229673 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229648 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229745 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229745 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229693 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whbs\" (UniqueName: \"kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229745 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229726 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.229745 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229741 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.229888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229758 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.229888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229827 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229848 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.229888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229879 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.230032 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229905 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.230032 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229967 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.230032 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.229990 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.230032 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.230004 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.239460 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.239442 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:54:33.239497 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.239444 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:54:33.239497 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:33.239488 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:33.239548 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:33.239526 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal" Oct 30 21:54:33.270365 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.270339 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjl7c\"" Oct 30 21:54:33.270876 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.270852 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:54:33.271261 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.271231 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hbp5d\"" Oct 30 21:54:33.271527 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.271505 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pgbrx\"" Oct 30 21:54:33.271600 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.271531 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4xfw\"" Oct 30 21:54:33.271905 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.271518 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.272053 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272036 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8jkh4\"" Oct 30 21:54:33.272230 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272211 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.272283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272231 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.272498 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272478 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.272601 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272569 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:54:33.272653 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272621 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:54:33.272873 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272855 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.272919 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272901 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4dnxg\"" Oct 30 21:54:33.273001 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272977 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:54:33.273146 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273068 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:54:33.273146 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.272855 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.273329 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273306 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:54:33.273417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273398 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:54:33.273523 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273509 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.273523 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273517 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:54:33.273660 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273625 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.273726 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273681 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s9wtv\"" Oct 30 21:54:33.273726 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273714 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66njg\"" Oct 30 21:54:33.273823 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273736 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.273823 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273733 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c7qtj\"" Oct 30 21:54:33.273823 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273800 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.273823 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273802 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:54:33.274009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273870 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ghldd\"" Oct 30 21:54:33.274009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273876 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:54:33.274009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273904 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:54:33.274009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273906 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.274009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.273917 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:54:33.274224 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.274068 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z7lww\"" Oct 30 21:54:33.274224 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.274181 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:54:33.274224 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.274182 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fd64m\"" Oct 30 21:54:33.275606 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275556 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.275606 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275598 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:54:33.275740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275598 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:54:33.275740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275623 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:54:33.275740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275702 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.275740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275720 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.275925 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.275805 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:54:33.279877 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.279857 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovn-node-metrics-cert\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.279877 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.279866 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-env-overrides\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.280090 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.279896 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-config\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.280090 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.279910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b6ed4c4-5043-4424-8b2d-826f478062c6-iptables-alerter-script\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.280202 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.280105 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86e6cf70-3f1a-40a1-b849-fcea928413c7-ovnkube-script-lib\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.280327 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.280309 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-original-pull-secret\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.296669 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.296652 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx44\" (UniqueName: \"kubernetes.io/projected/5b6ed4c4-5043-4424-8b2d-826f478062c6-kube-api-access-nxx44\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.305273 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.305257 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8d8\" (UniqueName: \"kubernetes.io/projected/b60aa182-3a90-40fd-b14d-620259e35523-kube-api-access-bw8d8\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.305531 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.305516 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whbs\" (UniqueName: \"kubernetes.io/projected/86e6cf70-3f1a-40a1-b849-fcea928413c7-kube-api-access-2whbs\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.313841 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.313825 4292 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:54:33.330514 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330495 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.330514 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330518 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.330667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330536 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.330667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330551 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.330667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330566 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.330667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330646 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-var-lib-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.330667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330663 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330680 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330732 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330735 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-dbus\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330795 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd-kubelet-config\") pod \"global-pull-secret-syncer-zrjkd\" (UID: \"4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd\") " pod="kube-system/global-pull-secret-syncer-zrjkd" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330820 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:33.330847 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330830 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-tmp\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330854 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330881 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330895 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330894 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f56f8924-c137-477d-a859-6defeffe45d4-serviceca\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330911 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330959 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-device-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330969 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.330995 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-bin\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331006 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331042 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xrf\" (UniqueName: \"kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331069 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331088 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331094 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a81c0006-0d2d-4e09-b222-99dd64093045-konnectivity-ca\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331122 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b6ed4c4-5043-4424-8b2d-826f478062c6-host-slash\") pod \"iptables-alerter-fcl4g\" (UID: \"5b6ed4c4-5043-4424-8b2d-826f478062c6\") " pod="openshift-network-operator/iptables-alerter-fcl4g" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331161 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331264 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331250 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-kubelet\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331297 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331323 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-node-log\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331341 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331364 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8426f\" (UniqueName: \"kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331402 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331427 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tv8\" (UniqueName: \"kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331500 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331530 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331546 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331561 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331627 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331640 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-tuned\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331714 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.331767 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331750 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331783 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331791 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-netns\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331798 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331818 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331842 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331857 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-multus-daemon-config\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331868 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331900 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331956 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-log-socket\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.331983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332019 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a81c0006-0d2d-4e09-b222-99dd64093045-agent-certs\") pod \"konnectivity-agent-mv9k2\" (UID: \"a81c0006-0d2d-4e09-b222-99dd64093045\") " pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332020 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332051 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332091 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332115 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332138 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332170 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332195 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332210 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332194 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.332255 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332253 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-ovn\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332287 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-etc-openvswitch\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332308 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332344 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332359 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-systemd-units\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332362 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-cni-netd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332390 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332435 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-slash\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332453 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-socket-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332496 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332523 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332547 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332595 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332624 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332659 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-registration-dir\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332680 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332715 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332760 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332799 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.332871 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332826 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wl4\" (UniqueName: \"kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332895 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332895 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332924 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-sys-fs\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332949 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.332982 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9ww\" (UniqueName: \"kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333010 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333035 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b60aa182-3a90-40fd-b14d-620259e35523-etc-selinux\") pod \"aws-ebs-csi-driver-node-gfjd6\" (UID: \"b60aa182-3a90-40fd-b14d-620259e35523\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gfjd6" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333076 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333125 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-tmp-dir\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333143 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw6w\" (UniqueName: \"kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333188 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333249 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333280 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333333 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333383 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333421 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86e6cf70-3f1a-40a1-b849-fcea928413c7-run-systemd\") pod \"ovnkube-node-4nsbz\" (UID: \"86e6cf70-3f1a-40a1-b849-fcea928413c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333460 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333478 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b39b57-4d19-426b-982b-4e197e37736f-cni-binary-copy\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.333698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.333494 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.393543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.393484 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tv8\" (UniqueName: \"kubernetes.io/projected/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-kube-api-access-w7tv8\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.393543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.393504 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8426f\" (UniqueName: \"kubernetes.io/projected/cf03a9d9-1567-4254-9747-d9836c323a2c-kube-api-access-8426f\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.393667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.393542 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xrf\" (UniqueName: \"kubernetes.io/projected/b1b39b57-4d19-426b-982b-4e197e37736f-kube-api-access-z8xrf\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.394880 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.394861 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wl4\" (UniqueName: \"kubernetes.io/projected/f56f8924-c137-477d-a859-6defeffe45d4-kube-api-access-47wl4\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.395080 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.395062 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw6w\" (UniqueName: \"kubernetes.io/projected/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-kube-api-access-8tw6w\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.395142 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.395114 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpcs\" (UniqueName: \"kubernetes.io/projected/685b08d4-b70f-4693-bced-4fd0b99d952a-kube-api-access-skpcs\") pod \"network-check-target-g2jrf\" (UID: \"685b08d4-b70f-4693-bced-4fd0b99d952a\") " pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:33.395178 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.395141 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9ww\" (UniqueName: \"kubernetes.io/projected/c10c2481-4327-49ef-a6b1-c922c6411099-kube-api-access-dg9ww\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.434097 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434076 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434153 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434153 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434115 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434153 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434129 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434143 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434172 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-multus-certs\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434186 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434209 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-bin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434213 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-kubernetes\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434226 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434244 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-var-lib-kubelet\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434257 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434226 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-host\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434263 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-kubelet\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434286 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f56f8924-c137-477d-a859-6defeffe45d4-host\") pod \"node-ca-4hdrj\" (UID: \"f56f8924-c137-477d-a859-6defeffe45d4\") " pod="openshift-image-registry/node-ca-4hdrj" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434294 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434309 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-system-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434348 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysconfig\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434312 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434401 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434431 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434442 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-socket-dir-parent\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434402 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434482 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434496 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-sys\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434504 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-run\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434512 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434532 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-var-lib-cni-multus\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434536 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434552 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434567 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434588 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-cnibin\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434613 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.434683 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434626 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-systemd\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434737 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434769 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434786 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434799 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434814 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434833 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434839 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-cni-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434839 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-cnibin\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434854 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434899 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-hostroot\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434902 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da87eda-28f6-4a7c-a004-42dd0b6a0d58-hosts-file\") pod \"node-resolver-x6pp9\" (UID: \"6da87eda-28f6-4a7c-a004-42dd0b6a0d58\") " pod="openshift-dns/node-resolver-x6pp9" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434896 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-netns\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434897 4292 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434932 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434938 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-modprobe-d\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434951 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-multus-conf-dir\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434977 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434983 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-host-run-k8s-cni-cncf-io\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.434917 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-lib-modules\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.435031 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.435083 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.435081 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf03a9d9-1567-4254-9747-d9836c323a2c-etc-sysctl-conf\") pod \"tuned-w9glb\" (UID: \"cf03a9d9-1567-4254-9747-d9836c323a2c\") " pod="openshift-cluster-node-tuning-operator/tuned-w9glb" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.435129 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1b39b57-4d19-426b-982b-4e197e37736f-etc-kubernetes\") pod \"multus-rmh74\" (UID: \"b1b39b57-4d19-426b-982b-4e197e37736f\") " pod="openshift-multus/multus-rmh74" Oct 30 21:54:33.435373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.435145 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc5a0a3-e851-4c62-8a13-0985bbb5068e-system-cni-dir\") pod \"multus-additional-cni-plugins-lgkpc\" (UID: \"cfc5a0a3-e851-4c62-8a13-0985bbb5068e\") " pod="openshift-multus/multus-additional-cni-plugins-lgkpc" Oct 30 21:54:33.436890 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.436875 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10c2481-4327-49ef-a6b1-c922c6411099-metrics-certs\") pod \"network-metrics-daemon-bqpgq\" (UID: \"c10c2481-4327-49ef-a6b1-c922c6411099\") " pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.514955 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.514930 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqpgq" Oct 30 21:54:33.766346 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.766278 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:54:33.924148 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.924120 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqpgq"] Oct 30 21:54:33.927422 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:33.927387 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10c2481_4327_49ef_a6b1_c922c6411099.slice/crio-72dd7addfe47a94b83e2b88cf22ed67e752959b83f919a91a23a930f0eac9203 WatchSource:0}: Error finding container 72dd7addfe47a94b83e2b88cf22ed67e752959b83f919a91a23a930f0eac9203: Status 404 returned error can't find the container with id 72dd7addfe47a94b83e2b88cf22ed67e752959b83f919a91a23a930f0eac9203 Oct 30 21:54:33.929376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:33.929362 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:54:34.172199 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:34.172173 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqpgq" event={"ID":"c10c2481-4327-49ef-a6b1-c922c6411099","Type":"ContainerStarted","Data":"72dd7addfe47a94b83e2b88cf22ed67e752959b83f919a91a23a930f0eac9203"} Oct 30 21:54:36.178280 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:36.178247 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqpgq" event={"ID":"c10c2481-4327-49ef-a6b1-c922c6411099","Type":"ContainerStarted","Data":"a3edd965f5bd5ac140e211e4c3c2eb9df0087c5d4a9e3bbb3a765b945b16f52a"} Oct 30 21:54:36.178738 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:36.178296 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqpgq" event={"ID":"c10c2481-4327-49ef-a6b1-c922c6411099","Type":"ContainerStarted","Data":"30a5b318030c4cd69d513313801dc2201cd3f76aabfe8ef3c0be163b87c118be"} Oct 30 21:54:38.448470 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:38.448440 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mv9k2" Oct 30 21:54:38.828677 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:38.828569 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:39.902750 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:39.902716 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:39.903236 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:39.902932 4292 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:54:43.320671 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.320637 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tqjdr"] Oct 30 21:54:43.322245 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.322230 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hfchg"] Oct 30 21:54:43.322363 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.322346 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.323818 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.323800 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.367808 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.367786 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xs6rg"] Oct 30 21:54:43.369485 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.369466 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.387731 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387709 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a898c28-096c-42ea-96fa-0a9600cadb07-tmp-dir\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.387819 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387737 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcd5d252-caa0-48b2-bceb-a675d13074d9-cert\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.387819 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387757 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxck\" (UniqueName: \"kubernetes.io/projected/dcd5d252-caa0-48b2-bceb-a675d13074d9-kube-api-access-vpxck\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.387819 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387776 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a898c28-096c-42ea-96fa-0a9600cadb07-metrics-tls\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.387921 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387819 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kb99\" (UniqueName: \"kubernetes.io/projected/4a898c28-096c-42ea-96fa-0a9600cadb07-kube-api-access-5kb99\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.387921 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.387898 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898c28-096c-42ea-96fa-0a9600cadb07-config-volume\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.406005 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.405984 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:54:43.406465 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.406441 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfchg"] Oct 30 21:54:43.407026 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.407005 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tqjdr"] Oct 30 21:54:43.407966 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.407938 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5khdr\"" Oct 30 21:54:43.407966 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.407953 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:54:43.418052 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.418026 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-j4tgb\"" Oct 30 21:54:43.418147 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.418100 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:54:43.418147 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.418117 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:54:43.456303 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.456251 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:54:43.456594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.456450 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:54:43.456594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.456485 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:54:43.456594 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.456559 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:54:43.458371 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.458347 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xs6rg"] Oct 30 21:54:43.461047 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.461027 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d4f564fcb-l8wjd"] Oct 30 21:54:43.462833 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.462819 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.488101 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488083 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a7baccb-3662-400c-851a-128e5ef76054-crio-socket\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.488194 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488108 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a7baccb-3662-400c-851a-128e5ef76054-data-volume\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.488194 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488128 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcd5d252-caa0-48b2-bceb-a675d13074d9-cert\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.488194 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488144 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxck\" (UniqueName: \"kubernetes.io/projected/dcd5d252-caa0-48b2-bceb-a675d13074d9-kube-api-access-vpxck\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.488377 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488357 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a898c28-096c-42ea-96fa-0a9600cadb07-metrics-tls\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.488453 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488439 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kb99\" (UniqueName: \"kubernetes.io/projected/4a898c28-096c-42ea-96fa-0a9600cadb07-kube-api-access-5kb99\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.488560 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488543 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a7baccb-3662-400c-851a-128e5ef76054-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.488621 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488596 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a7baccb-3662-400c-851a-128e5ef76054-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.488659 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488637 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898c28-096c-42ea-96fa-0a9600cadb07-config-volume\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.488698 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488689 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a898c28-096c-42ea-96fa-0a9600cadb07-tmp-dir\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.488748 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488712 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppfm\" (UniqueName: \"kubernetes.io/projected/0a7baccb-3662-400c-851a-128e5ef76054-kube-api-access-zppfm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.489017 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.488999 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a898c28-096c-42ea-96fa-0a9600cadb07-tmp-dir\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.489145 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.489128 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898c28-096c-42ea-96fa-0a9600cadb07-config-volume\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.490556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.490538 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a898c28-096c-42ea-96fa-0a9600cadb07-metrics-tls\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.490762 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.490741 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcd5d252-caa0-48b2-bceb-a675d13074d9-cert\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.505998 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.505978 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-plmg6\"" Oct 30 21:54:43.506096 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.506082 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:54:43.568294 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.568257 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:54:43.568630 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.568610 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4f564fcb-l8wjd"] Oct 30 21:54:43.570160 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.570140 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kb99\" (UniqueName: \"kubernetes.io/projected/4a898c28-096c-42ea-96fa-0a9600cadb07-kube-api-access-5kb99\") pod \"dns-default-hfchg\" (UID: \"4a898c28-096c-42ea-96fa-0a9600cadb07\") " pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.570244 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.570183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxck\" (UniqueName: \"kubernetes.io/projected/dcd5d252-caa0-48b2-bceb-a675d13074d9-kube-api-access-vpxck\") pod \"ingress-canary-tqjdr\" (UID: \"dcd5d252-caa0-48b2-bceb-a675d13074d9\") " pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.589283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589230 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppfm\" (UniqueName: \"kubernetes.io/projected/0a7baccb-3662-400c-851a-128e5ef76054-kube-api-access-zppfm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589283 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589279 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-certificates\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589307 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a7baccb-3662-400c-851a-128e5ef76054-crio-socket\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589326 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a7baccb-3662-400c-851a-128e5ef76054-data-volume\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589340 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-ca-trust-extracted\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589355 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-trusted-ca\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589372 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw868\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-kube-api-access-nw868\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589394 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-tls\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589623 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589413 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a7baccb-3662-400c-851a-128e5ef76054-crio-socket\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589623 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589514 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-image-registry-private-configuration\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589623 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589536 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-installation-pull-secrets\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589714 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589635 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a7baccb-3662-400c-851a-128e5ef76054-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589714 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589671 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a7baccb-3662-400c-851a-128e5ef76054-data-volume\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.589714 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589672 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-bound-sa-token\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.589802 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.589711 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a7baccb-3662-400c-851a-128e5ef76054-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.590227 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.590211 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a7baccb-3662-400c-851a-128e5ef76054-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.591695 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.591678 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a7baccb-3662-400c-851a-128e5ef76054-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.606539 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.606511 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bmck6\"" Oct 30 21:54:43.606539 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.606516 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:54:43.606688 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.606533 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:54:43.606688 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.606593 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:54:43.632740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.632720 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tqjdr" Oct 30 21:54:43.639049 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.639027 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:43.653186 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.653162 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppfm\" (UniqueName: \"kubernetes.io/projected/0a7baccb-3662-400c-851a-128e5ef76054-kube-api-access-zppfm\") pod \"insights-runtime-extractor-xs6rg\" (UID: \"0a7baccb-3662-400c-851a-128e5ef76054\") " pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.681498 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.681472 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xs6rg" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691278 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-image-registry-private-configuration\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691322 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-installation-pull-secrets\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-bound-sa-token\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691498 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-certificates\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691536 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-ca-trust-extracted\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691561 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-trusted-ca\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691668 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw868\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-kube-api-access-nw868\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.691770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.691696 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-tls\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.693892 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.693868 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-certificates\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.695292 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.694605 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-ca-trust-extracted\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.695292 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.695257 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-trusted-ca\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.696854 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.696829 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-installation-pull-secrets\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.700854 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.700808 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-image-registry-private-configuration\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.701869 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.701843 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-registry-tls\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.758453 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.758405 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-bound-sa-token\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.758550 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.758490 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw868\" (UniqueName: \"kubernetes.io/projected/5abdffd0-5ef3-4b12-89ef-f69f012d4ffe-kube-api-access-nw868\") pod \"image-registry-6d4f564fcb-l8wjd\" (UID: \"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe\") " pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:43.771490 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:43.771466 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:44.049272 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.049245 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tqjdr"] Oct 30 21:54:44.051724 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:44.051697 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd5d252_caa0_48b2_bceb_a675d13074d9.slice/crio-3e35a30ee9f5d8a2d92fcc8cd625c7344b78701f32a5087d2e942a86863de791 WatchSource:0}: Error finding container 3e35a30ee9f5d8a2d92fcc8cd625c7344b78701f32a5087d2e942a86863de791: Status 404 returned error can't find the container with id 3e35a30ee9f5d8a2d92fcc8cd625c7344b78701f32a5087d2e942a86863de791 Oct 30 21:54:44.055007 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.054983 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfchg"] Oct 30 21:54:44.057309 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:44.057289 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a898c28_096c_42ea_96fa_0a9600cadb07.slice/crio-8ba2159ca566dd3752d64341e1b0341da2514ad037f96e4bde3dd68c3e1fadbc WatchSource:0}: Error finding container 8ba2159ca566dd3752d64341e1b0341da2514ad037f96e4bde3dd68c3e1fadbc: Status 404 returned error can't find the container with id 8ba2159ca566dd3752d64341e1b0341da2514ad037f96e4bde3dd68c3e1fadbc Oct 30 21:54:44.107896 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.107869 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xs6rg"] Oct 30 21:54:44.110458 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:44.110433 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7baccb_3662_400c_851a_128e5ef76054.slice/crio-870493b2a6e5f0ec6582a831c6414b0046d18ecb0b457d4011df961032a14fce WatchSource:0}: Error finding container 870493b2a6e5f0ec6582a831c6414b0046d18ecb0b457d4011df961032a14fce: Status 404 returned error can't find the container with id 870493b2a6e5f0ec6582a831c6414b0046d18ecb0b457d4011df961032a14fce Oct 30 21:54:44.158776 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.158753 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4f564fcb-l8wjd"] Oct 30 21:54:44.161690 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:44.161668 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abdffd0_5ef3_4b12_89ef_f69f012d4ffe.slice/crio-9b25b126bcfb8d834dea0403ed1341dee9bd95f818ad0cbd7b8ad01fcbabf637 WatchSource:0}: Error finding container 9b25b126bcfb8d834dea0403ed1341dee9bd95f818ad0cbd7b8ad01fcbabf637: Status 404 returned error can't find the container with id 9b25b126bcfb8d834dea0403ed1341dee9bd95f818ad0cbd7b8ad01fcbabf637 Oct 30 21:54:44.195213 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.195112 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfchg" event={"ID":"4a898c28-096c-42ea-96fa-0a9600cadb07","Type":"ContainerStarted","Data":"8ba2159ca566dd3752d64341e1b0341da2514ad037f96e4bde3dd68c3e1fadbc"} Oct 30 21:54:44.196363 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.196335 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tqjdr" event={"ID":"dcd5d252-caa0-48b2-bceb-a675d13074d9","Type":"ContainerStarted","Data":"3e35a30ee9f5d8a2d92fcc8cd625c7344b78701f32a5087d2e942a86863de791"} Oct 30 21:54:44.197763 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.197736 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs6rg" event={"ID":"0a7baccb-3662-400c-851a-128e5ef76054","Type":"ContainerStarted","Data":"712baacb38a918a12ce1c335e70d6f307cee7243092348826c5ec0ad4374f2d2"} Oct 30 21:54:44.197859 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.197783 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs6rg" event={"ID":"0a7baccb-3662-400c-851a-128e5ef76054","Type":"ContainerStarted","Data":"870493b2a6e5f0ec6582a831c6414b0046d18ecb0b457d4011df961032a14fce"} Oct 30 21:54:44.198689 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:44.198668 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" event={"ID":"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe","Type":"ContainerStarted","Data":"9b25b126bcfb8d834dea0403ed1341dee9bd95f818ad0cbd7b8ad01fcbabf637"} Oct 30 21:54:45.203742 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:45.203697 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" event={"ID":"5abdffd0-5ef3-4b12-89ef-f69f012d4ffe","Type":"ContainerStarted","Data":"663ed903193448b8255d28572afdbed7df0a5eadf9842aaa87a944aed2389480"} Oct 30 21:54:45.204458 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:45.204393 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:54:46.210769 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:46.210731 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs6rg" event={"ID":"0a7baccb-3662-400c-851a-128e5ef76054","Type":"ContainerStarted","Data":"c9a66b2a8ef1a9c6014d44a9d1ea892596b0a9c334901743b3a42033b6da8016"} Oct 30 21:54:47.216897 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:47.216359 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfchg" event={"ID":"4a898c28-096c-42ea-96fa-0a9600cadb07","Type":"ContainerStarted","Data":"c28e76b0104eb1847907afac017723466f57052f0913fac941885c34ec687d8e"} Oct 30 21:54:47.216897 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:47.216455 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfchg" event={"ID":"4a898c28-096c-42ea-96fa-0a9600cadb07","Type":"ContainerStarted","Data":"847cc60c779d6ac0ee9630f2768702dc2c318d278f1c63d2c0bff91554bb1aaf"} Oct 30 21:54:47.216897 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:47.216817 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:47.341144 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:47.341089 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hfchg" podStartSLOduration=2.073598896 podStartE2EDuration="4.341070736s" totalImagesPullingTime="2.26747184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:47.3409585 +0000 UTC m=+15.309778066" watchObservedRunningTime="2026-10-30 21:54:47.341070736 +0000 UTC m=+15.309890296" Oct 30 21:54:47.341340 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:47.341226 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" podStartSLOduration=4.341220795 podStartE2EDuration="4.341220795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:45.328954798 +0000 UTC m=+13.297774362" watchObservedRunningTime="2026-10-30 21:54:47.341220795 +0000 UTC m=+15.310040359" Oct 30 21:54:48.221130 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:48.221090 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tqjdr" event={"ID":"dcd5d252-caa0-48b2-bceb-a675d13074d9","Type":"ContainerStarted","Data":"f7a41cba29dc38686d962bb7627a3177a51fab96d343f88119d889c09b75d74f"} Oct 30 21:54:48.223036 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:48.223008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs6rg" event={"ID":"0a7baccb-3662-400c-851a-128e5ef76054","Type":"ContainerStarted","Data":"140d858a310685422d7b48a51a745df200f0d2f0588141562661e507b78b59b4"} Oct 30 21:54:48.344705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:48.344659 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tqjdr" podStartSLOduration=2.134883004 podStartE2EDuration="5.344646482s" totalImagesPullingTime="3.209763478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:48.344393051 +0000 UTC m=+16.313212614" watchObservedRunningTime="2026-10-30 21:54:48.344646482 +0000 UTC m=+16.313466045" Oct 30 21:54:48.467236 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:48.467188 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xs6rg" podStartSLOduration=2.29263754 podStartE2EDuration="5.467175134s" totalImagesPullingTime="3.174537594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:48.466635762 +0000 UTC m=+16.435455327" watchObservedRunningTime="2026-10-30 21:54:48.467175134 +0000 UTC m=+16.435994694" Oct 30 21:54:48.805704 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:48.805675 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nsbz" Oct 30 21:54:49.045175 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.045136 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fk4dv"] Oct 30 21:54:49.046884 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.046862 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.075079 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.075024 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fk4dv"] Oct 30 21:54:49.102900 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.102878 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:54:49.103017 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.102920 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:54:49.103064 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.103052 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-cqdwr\"" Oct 30 21:54:49.128976 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.128957 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt66l\" (UniqueName: \"kubernetes.io/projected/e1ba98f1-61eb-4670-8a92-b93ac26726ce-kube-api-access-vt66l\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.129034 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.128984 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ba98f1-61eb-4670-8a92-b93ac26726ce-tmp\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.229324 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.229302 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ba98f1-61eb-4670-8a92-b93ac26726ce-tmp\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.229640 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.229344 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt66l\" (UniqueName: \"kubernetes.io/projected/e1ba98f1-61eb-4670-8a92-b93ac26726ce-kube-api-access-vt66l\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.229640 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.229605 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ba98f1-61eb-4670-8a92-b93ac26726ce-tmp\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.292294 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.292274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt66l\" (UniqueName: \"kubernetes.io/projected/e1ba98f1-61eb-4670-8a92-b93ac26726ce-kube-api-access-vt66l\") pod \"downloads-6dd789c8b7-fk4dv\" (UID: \"e1ba98f1-61eb-4670-8a92-b93ac26726ce\") " pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.356418 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.356359 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:54:49.776101 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:49.776032 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fk4dv"] Oct 30 21:54:49.778836 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:49.778806 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ba98f1_61eb_4670_8a92_b93ac26726ce.slice/crio-c06783abe6ae85024ee439c37eb1f2bf75456afc23f37e57dfdf108875dbb136 WatchSource:0}: Error finding container c06783abe6ae85024ee439c37eb1f2bf75456afc23f37e57dfdf108875dbb136: Status 404 returned error can't find the container with id c06783abe6ae85024ee439c37eb1f2bf75456afc23f37e57dfdf108875dbb136 Oct 30 21:54:50.230047 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:50.230013 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fk4dv" event={"ID":"e1ba98f1-61eb-4670-8a92-b93ac26726ce","Type":"ContainerStarted","Data":"c06783abe6ae85024ee439c37eb1f2bf75456afc23f37e57dfdf108875dbb136"} Oct 30 21:54:52.357692 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.357652 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q"] Oct 30 21:54:52.359849 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.359823 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:52.451156 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.451117 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6fd4e3b-b4ef-4693-a404-5fa82006825b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jm8q\" (UID: \"e6fd4e3b-b4ef-4693-a404-5fa82006825b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:52.467911 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.467885 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lppft\"" Oct 30 21:54:52.467911 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.467915 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:54:52.468727 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.468702 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q"] Oct 30 21:54:52.552349 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.552323 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6fd4e3b-b4ef-4693-a404-5fa82006825b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jm8q\" (UID: \"e6fd4e3b-b4ef-4693-a404-5fa82006825b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:52.554827 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.554805 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6fd4e3b-b4ef-4693-a404-5fa82006825b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jm8q\" (UID: \"e6fd4e3b-b4ef-4693-a404-5fa82006825b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:52.671271 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:52.671191 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:53.058076 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:53.058042 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q"] Oct 30 21:54:53.063771 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:53.063722 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd4e3b_b4ef_4693_a404_5fa82006825b.slice/crio-4e83cd128134b16e223fb1834473ed19f546db17eae9d33fba2fb7c086a881cb WatchSource:0}: Error finding container 4e83cd128134b16e223fb1834473ed19f546db17eae9d33fba2fb7c086a881cb: Status 404 returned error can't find the container with id 4e83cd128134b16e223fb1834473ed19f546db17eae9d33fba2fb7c086a881cb Oct 30 21:54:53.245282 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:53.245241 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" event={"ID":"e6fd4e3b-b4ef-4693-a404-5fa82006825b","Type":"ContainerStarted","Data":"4e83cd128134b16e223fb1834473ed19f546db17eae9d33fba2fb7c086a881cb"} Oct 30 21:54:55.254391 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:55.254348 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" event={"ID":"e6fd4e3b-b4ef-4693-a404-5fa82006825b","Type":"ContainerStarted","Data":"6306d387f55952939d38ea127b0740f49c26619449d80362814161af33344335"} Oct 30 21:54:55.254819 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:55.254560 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:55.259927 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:55.259905 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" Oct 30 21:54:55.373336 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:55.373288 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jm8q" podStartSLOduration=1.979361109 podStartE2EDuration="3.373277377s" totalImagesPullingTime="1.393916268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:55.372793978 +0000 UTC m=+23.341613543" watchObservedRunningTime="2026-10-30 21:54:55.373277377 +0000 UTC m=+23.342096940" Oct 30 21:54:56.642382 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:54:56.642343 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:54:56.957244 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:56.957167 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-598l5"] Oct 30 21:54:56.960140 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:56.960111 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.064690 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.064664 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:54:57.064690 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.064690 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:54:57.064892 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.064763 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sshcp\"" Oct 30 21:54:57.065685 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.065667 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:54:57.079057 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.079029 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht7f\" (UniqueName: \"kubernetes.io/projected/a855eb26-b20b-452c-826d-73a8263ef31d-kube-api-access-2ht7f\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.079169 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.079083 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a855eb26-b20b-452c-826d-73a8263ef31d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.079301 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.079238 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.079348 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.079319 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.106337 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.106315 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:54:57.106449 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.106352 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:54:57.116804 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.116776 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-598l5"] Oct 30 21:54:57.180486 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.180455 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.180596 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.180497 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.180596 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.180531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht7f\" (UniqueName: \"kubernetes.io/projected/a855eb26-b20b-452c-826d-73a8263ef31d-kube-api-access-2ht7f\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.180596 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.180584 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a855eb26-b20b-452c-826d-73a8263ef31d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.181323 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.181283 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a855eb26-b20b-452c-826d-73a8263ef31d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.182965 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.182941 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.183072 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.182993 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a855eb26-b20b-452c-826d-73a8263ef31d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.244944 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.244883 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht7f\" (UniqueName: \"kubernetes.io/projected/a855eb26-b20b-452c-826d-73a8263ef31d-kube-api-access-2ht7f\") pod \"prometheus-operator-9cd89b74c-598l5\" (UID: \"a855eb26-b20b-452c-826d-73a8263ef31d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.273403 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.273381 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" Oct 30 21:54:57.648509 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:57.648477 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-598l5"] Oct 30 21:54:57.652051 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:54:57.652018 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda855eb26_b20b_452c_826d_73a8263ef31d.slice/crio-5e8c63da1034d1a67ee143b43d060ee432f86e7459f3bbdd3d5841eab6d84217 WatchSource:0}: Error finding container 5e8c63da1034d1a67ee143b43d060ee432f86e7459f3bbdd3d5841eab6d84217: Status 404 returned error can't find the container with id 5e8c63da1034d1a67ee143b43d060ee432f86e7459f3bbdd3d5841eab6d84217 Oct 30 21:54:58.229753 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:58.229719 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hfchg" Oct 30 21:54:58.265789 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:54:58.265754 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" event={"ID":"a855eb26-b20b-452c-826d-73a8263ef31d","Type":"ContainerStarted","Data":"5e8c63da1034d1a67ee143b43d060ee432f86e7459f3bbdd3d5841eab6d84217"} Oct 30 21:55:00.274212 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:00.274176 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" event={"ID":"a855eb26-b20b-452c-826d-73a8263ef31d","Type":"ContainerStarted","Data":"a1cc5026a15f629d1a2436c3c15c5ea27291b05a031312b5d12314a2a988eed6"} Oct 30 21:55:00.274669 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:00.274229 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" event={"ID":"a855eb26-b20b-452c-826d-73a8263ef31d","Type":"ContainerStarted","Data":"ccc93bd7a91178f881b435259f0813a04ccd216c6f3245ddc74bab3cbfe08a56"} Oct 30 21:55:00.408094 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:00.408036 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-598l5" podStartSLOduration=2.788036066 podStartE2EDuration="4.4080184s" totalImagesPullingTime="1.619982334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:00.407312441 +0000 UTC m=+28.376132006" watchObservedRunningTime="2026-10-30 21:55:00.4080184 +0000 UTC m=+28.376837969" Oct 30 21:55:00.944916 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:00.944884 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:00.946991 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:00.946964 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.002009 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.001981 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:55:01.011261 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011226 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011261 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011261 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011311 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011342 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011382 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cltw\" (UniqueName: \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011407 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.011438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.011435 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.050959 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.050906 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:55:01.051091 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.051019 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:55:01.051091 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.051035 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:55:01.051417 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.051393 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wwkpf\"" Oct 30 21:55:01.051907 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.051886 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:01.052559 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.052543 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:55:01.112169 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112126 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112169 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112168 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112208 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112231 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112260 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cltw\" (UniqueName: \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112284 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112376 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112324 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.112709 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.112686 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.113077 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.113051 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.113194 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.113101 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.113194 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.113133 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.115007 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.114985 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.115366 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.115347 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.178529 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.178494 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cltw\" (UniqueName: \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\") pod \"console-f44b89489-fwk5c\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.258317 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.258236 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:01.638352 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:01.638311 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:01.642215 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:55:01.642179 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f329eb_8a5a_445d_8db6_3203b29116f3.slice/crio-db3db6f89d2eb83d333ba7923e82d08e1774696a40cfc8686eba7f0088352783 WatchSource:0}: Error finding container db3db6f89d2eb83d333ba7923e82d08e1774696a40cfc8686eba7f0088352783: Status 404 returned error can't find the container with id db3db6f89d2eb83d333ba7923e82d08e1774696a40cfc8686eba7f0088352783 Oct 30 21:55:02.284221 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:02.284182 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f44b89489-fwk5c" event={"ID":"68f329eb-8a5a-445d-8db6-3203b29116f3","Type":"ContainerStarted","Data":"db3db6f89d2eb83d333ba7923e82d08e1774696a40cfc8686eba7f0088352783"} Oct 30 21:55:03.701851 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.701817 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4"] Oct 30 21:55:03.719827 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.719798 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.722428 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.722397 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s4nn9"] Oct 30 21:55:03.739840 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.739819 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.775212 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.775185 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4"] Oct 30 21:55:03.780965 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.780945 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:55:03.829852 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.829829 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:55:03.829852 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.829828 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:55:03.830028 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.829858 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:55:03.830028 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.829867 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g9hfl\"" Oct 30 21:55:03.830028 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.829916 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-szz58\"" Oct 30 21:55:03.833738 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.830208 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:55:03.834456 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834422 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.834556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834470 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.834657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834568 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834657 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834639 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-wtmp\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834749 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834724 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-tls\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834789 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834762 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.834789 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834783 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-sys\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834886 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834830 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8xb\" (UniqueName: \"kubernetes.io/projected/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-api-access-dx8xb\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.834922 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834902 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-textfile\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834954 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834924 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-metrics-client-ca\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.834954 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834941 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.835035 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.834966 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnv7w\" (UniqueName: \"kubernetes.io/projected/be21030f-b519-4fef-9167-440ba1615ff8-kube-api-access-wnv7w\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.835080 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.835027 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.835124 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.835106 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/405e081e-ac16-4ae4-8050-ba36e14b2d32-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.835162 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.835123 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-root\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.880787 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.880757 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935773 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935830 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-wtmp\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935871 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-tls\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935904 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935931 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-sys\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935958 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8xb\" (UniqueName: \"kubernetes.io/projected/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-api-access-dx8xb\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.935996 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-textfile\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936023 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-metrics-client-ca\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936051 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936077 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnv7w\" (UniqueName: \"kubernetes.io/projected/be21030f-b519-4fef-9167-440ba1615ff8-kube-api-access-wnv7w\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936107 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936138 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/405e081e-ac16-4ae4-8050-ba36e14b2d32-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936164 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-root\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936192 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.936651 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.936225 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.937496 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.937148 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-metrics-client-ca\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.937496 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.937292 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-wtmp\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.937761 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.937846 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-root\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938283 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-textfile\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938334 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be21030f-b519-4fef-9167-440ba1615ff8-sys\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938591 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938605 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938655 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/405e081e-ac16-4ae4-8050-ba36e14b2d32-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.938817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.938780 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/405e081e-ac16-4ae4-8050-ba36e14b2d32-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.939976 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.939954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:03.940064 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.939960 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be21030f-b519-4fef-9167-440ba1615ff8-node-exporter-tls\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:03.940389 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:03.940366 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:04.001771 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.001709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnv7w\" (UniqueName: \"kubernetes.io/projected/be21030f-b519-4fef-9167-440ba1615ff8-kube-api-access-wnv7w\") pod \"node-exporter-s4nn9\" (UID: \"be21030f-b519-4fef-9167-440ba1615ff8\") " pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:04.001931 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.001911 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8xb\" (UniqueName: \"kubernetes.io/projected/405e081e-ac16-4ae4-8050-ba36e14b2d32-kube-api-access-dx8xb\") pod \"kube-state-metrics-5dc8d7d89b-2sxz4\" (UID: \"405e081e-ac16-4ae4-8050-ba36e14b2d32\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:04.075389 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.075350 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" Oct 30 21:55:04.118992 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.118965 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s4nn9" Oct 30 21:55:04.177227 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.175124 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g2jrf" Oct 30 21:55:04.294428 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.294343 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4nn9" event={"ID":"be21030f-b519-4fef-9167-440ba1615ff8","Type":"ContainerStarted","Data":"ffa17ce9f48569505d82f6cc1918412598c33dfdc3cfbc71bf6f1d5650a5c699"} Oct 30 21:55:04.529095 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:04.529064 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4"] Oct 30 21:55:04.532027 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:55:04.531987 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405e081e_ac16_4ae4_8050_ba36e14b2d32.slice/crio-d5816fb1b319034c9c7b827caf3e7ecc6c605036ff68a6de66b74d1f29ecf599 WatchSource:0}: Error finding container d5816fb1b319034c9c7b827caf3e7ecc6c605036ff68a6de66b74d1f29ecf599: Status 404 returned error can't find the container with id d5816fb1b319034c9c7b827caf3e7ecc6c605036ff68a6de66b74d1f29ecf599 Oct 30 21:55:05.299090 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:05.299045 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" event={"ID":"405e081e-ac16-4ae4-8050-ba36e14b2d32","Type":"ContainerStarted","Data":"d5816fb1b319034c9c7b827caf3e7ecc6c605036ff68a6de66b74d1f29ecf599"} Oct 30 21:55:06.307083 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.307044 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f44b89489-fwk5c" event={"ID":"68f329eb-8a5a-445d-8db6-3203b29116f3","Type":"ContainerStarted","Data":"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368"} Oct 30 21:55:06.412346 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.412309 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-588b9668b8-xsbwg"] Oct 30 21:55:06.414563 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.414536 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.462530 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.462480 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f44b89489-fwk5c" podStartSLOduration=2.555701037 podStartE2EDuration="6.462464748s" totalImagesPullingTime="3.906763711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:06.462102559 +0000 UTC m=+34.430922150" watchObservedRunningTime="2026-10-30 21:55:06.462464748 +0000 UTC m=+34.431284358" Oct 30 21:55:06.479655 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.479510 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-588b9668b8-xsbwg"] Oct 30 21:55:06.533249 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533210 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:55:06.533526 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533473 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:55:06.533526 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533492 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-574cm940a5b0p\"" Oct 30 21:55:06.533526 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533518 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-htw5r\"" Oct 30 21:55:06.533705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533524 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:55:06.533705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533551 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:55:06.533705 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.533552 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:55:06.561679 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561614 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561679 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561650 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e139ebf4-48aa-491c-b4e8-70c64845dee9-tmp\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561679 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561697 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561712 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-grpc-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561729 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561748 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561764 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e139ebf4-48aa-491c-b4e8-70c64845dee9-metrics-client-ca\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.561848 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.561780 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxf84\" (UniqueName: \"kubernetes.io/projected/e139ebf4-48aa-491c-b4e8-70c64845dee9-kube-api-access-pxf84\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662469 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662518 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-grpc-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662776 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662818 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662845 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e139ebf4-48aa-491c-b4e8-70c64845dee9-metrics-client-ca\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662872 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxf84\" (UniqueName: \"kubernetes.io/projected/e139ebf4-48aa-491c-b4e8-70c64845dee9-kube-api-access-pxf84\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662955 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.662986 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e139ebf4-48aa-491c-b4e8-70c64845dee9-tmp\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663098 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.663020 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663680 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.663503 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e139ebf4-48aa-491c-b4e8-70c64845dee9-tmp\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.663718 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.663700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e139ebf4-48aa-491c-b4e8-70c64845dee9-metrics-client-ca\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.666756 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.666703 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.666756 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.666746 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-grpc-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.666930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.666828 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.667971 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.667950 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.668239 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.668219 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-tls\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.668294 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.668241 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e139ebf4-48aa-491c-b4e8-70c64845dee9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.727474 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.727449 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxf84\" (UniqueName: \"kubernetes.io/projected/e139ebf4-48aa-491c-b4e8-70c64845dee9-kube-api-access-pxf84\") pod \"thanos-querier-588b9668b8-xsbwg\" (UID: \"e139ebf4-48aa-491c-b4e8-70c64845dee9\") " pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:06.734812 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:06.734788 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:07.179477 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.179390 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-588b9668b8-xsbwg"] Oct 30 21:55:07.183244 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:55:07.183209 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode139ebf4_48aa_491c_b4e8_70c64845dee9.slice/crio-af7d65bb47b4913ce7f1c35b5e9fc47cb355d80e3b0b3aba82bb348863a1cae3 WatchSource:0}: Error finding container af7d65bb47b4913ce7f1c35b5e9fc47cb355d80e3b0b3aba82bb348863a1cae3: Status 404 returned error can't find the container with id af7d65bb47b4913ce7f1c35b5e9fc47cb355d80e3b0b3aba82bb348863a1cae3 Oct 30 21:55:07.222060 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.222030 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d4f564fcb-l8wjd" Oct 30 21:55:07.313876 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.313831 4292 generic.go:356] "Generic (PLEG): container finished" podID="be21030f-b519-4fef-9167-440ba1615ff8" containerID="99034ad15a7b00fe203466a0595b34fbb53c1c749947d30861c1e2cf706a81db" exitCode=0 Oct 30 21:55:07.314360 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.313940 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4nn9" event={"ID":"be21030f-b519-4fef-9167-440ba1615ff8","Type":"ContainerDied","Data":"99034ad15a7b00fe203466a0595b34fbb53c1c749947d30861c1e2cf706a81db"} Oct 30 21:55:07.315549 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.315523 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"af7d65bb47b4913ce7f1c35b5e9fc47cb355d80e3b0b3aba82bb348863a1cae3"} Oct 30 21:55:07.322616 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.322470 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" event={"ID":"405e081e-ac16-4ae4-8050-ba36e14b2d32","Type":"ContainerStarted","Data":"19184a56fca69a2aa5122ab51f0060d252ebc8d3f16bb01bdb9bf49631c67101"} Oct 30 21:55:07.322616 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.322525 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" event={"ID":"405e081e-ac16-4ae4-8050-ba36e14b2d32","Type":"ContainerStarted","Data":"e5ce87449c20871a4c0fc59ab205fc0fffaa6e96aba25c9f86ff6130b0afcb24"} Oct 30 21:55:07.322616 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.322555 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" event={"ID":"405e081e-ac16-4ae4-8050-ba36e14b2d32","Type":"ContainerStarted","Data":"fe41f16624fd7629dcee9d03c086975555c364a7e2a985b5712e8f51d0a5a429"} Oct 30 21:55:07.324952 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.324867 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fk4dv" event={"ID":"e1ba98f1-61eb-4670-8a92-b93ac26726ce","Type":"ContainerStarted","Data":"d76c3a1e5596ac8ca6b40abd2490f339ffd03f37f319f48f0fdad5e02be85219"} Oct 30 21:55:07.325252 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.325204 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:55:07.344907 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.340131 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fk4dv" Oct 30 21:55:07.492469 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.492420 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2sxz4" podStartSLOduration=2.372504689 podStartE2EDuration="4.492405137s" totalImagesPullingTime="2.119900448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:07.492005841 +0000 UTC m=+35.460825406" watchObservedRunningTime="2026-10-30 21:55:07.492405137 +0000 UTC m=+35.461224703" Oct 30 21:55:07.737856 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:07.737749 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fk4dv" podStartSLOduration=2.682382816 podStartE2EDuration="19.737732573s" totalImagesPullingTime="17.055349757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:07.737728698 +0000 UTC m=+35.706548266" watchObservedRunningTime="2026-10-30 21:55:07.737732573 +0000 UTC m=+35.706552138" Oct 30 21:55:08.333881 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:08.333751 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4nn9" event={"ID":"be21030f-b519-4fef-9167-440ba1615ff8","Type":"ContainerStarted","Data":"365b67f667f1420d85ba46aa5e5e8df46303a8fad916d138e43e9ed9da53f0fa"} Oct 30 21:55:08.333881 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:08.333839 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4nn9" event={"ID":"be21030f-b519-4fef-9167-440ba1615ff8","Type":"ContainerStarted","Data":"fa1a5c9f5213d4b218ab2b72196bfed9e9f57a12db4dfd245a968c12ca1c0b51"} Oct 30 21:55:11.258748 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.258654 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:11.258748 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.258718 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:11.264330 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.264307 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:11.346121 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.346086 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"fbc4c0221d89d8a13d5bef4cbe80b6366c0d73c44ae0f4083fc592e2673ba90a"} Oct 30 21:55:11.346263 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.346179 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"36debe375438ce55c09f20852a5bf5fa813d6a672b7483518da6b8575af0453a"} Oct 30 21:55:11.346263 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.346222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"7bb854f0ec63b0f54f30ee927a0856b32164783ba26b4d616d6a46e20e420af1"} Oct 30 21:55:11.350342 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.350315 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:11.385335 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:11.385284 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s4nn9" podStartSLOduration=6.043949537 podStartE2EDuration="8.385272273s" totalImagesPullingTime="2.260475218s" totalInitContainerRuntime="80.847518ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:08.46273596 +0000 UTC m=+36.431555535" watchObservedRunningTime="2026-10-30 21:55:11.385272273 +0000 UTC m=+39.354091836" Oct 30 21:55:13.290337 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.290252 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:55:13.312438 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.312411 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.358676 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.358641 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:55:13.408269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.408239 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:55:13.409235 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.409211 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:55:13.409361 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.409274 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:55:13.409410 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.409365 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jcc4t\"" Oct 30 21:55:13.409460 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.409411 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:55:13.409505 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.409500 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:55:13.425526 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425497 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425661 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425587 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425661 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425642 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425786 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425708 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425786 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425753 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425890 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425818 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.425944 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425910 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426003 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425947 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426003 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425972 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426003 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.425998 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426134 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426037 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426134 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426115 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khfm\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426231 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426150 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426231 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426204 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426333 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426233 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426333 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426444 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426349 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426444 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426389 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426444 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426412 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.426593 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.426444 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.460073 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.460037 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"dfdf87e1727416578f29ceb081509a6bdd46b1182b3fbe1db4940ff8039c77c5"} Oct 30 21:55:13.460221 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.460113 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"431a5a28242c7847e1b3fbac3fb572094bf29a8581ca0876f970951599d513d6"} Oct 30 21:55:13.460221 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.460156 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" event={"ID":"e139ebf4-48aa-491c-b4e8-70c64845dee9","Type":"ContainerStarted","Data":"6a649631c5ca5ecb97eb05e8653e80bf8f736fa1fffd38708b9bde8e95e0c9dc"} Oct 30 21:55:13.460331 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.460294 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:13.474162 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474138 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:55:13.474296 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474282 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:55:13.474369 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474327 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:55:13.474369 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474346 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:55:13.474479 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474361 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9v1fbagc363of\"" Oct 30 21:55:13.474479 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474384 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:55:13.474699 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474681 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:55:13.474784 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474763 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:55:13.474841 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.474830 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:55:13.525922 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.525902 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:55:13.527184 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527158 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527284 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527284 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527284 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527268 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527291 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527316 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527344 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527374 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527399 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527443 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527426 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527451 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527480 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khfm\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527507 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527533 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527556 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527612 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527642 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527686 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527714 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.527810 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527748 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.528294 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.527854 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.528352 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.528304 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.529535 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.529508 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.529828 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.529801 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.529919 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.529890 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.530545 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.530516 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.532008 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.531976 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.532422 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.532383 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.533059 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.532743 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.535065 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.535039 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.535397 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.535081 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.535397 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.535180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.535518 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.535421 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.536278 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.536244 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.536620 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.536597 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.536853 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.536837 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.537664 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.537645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.539979 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.539954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.561886 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.561866 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:55:13.572527 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.572432 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.594538 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.594518 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khfm\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\") pod \"prometheus-k8s-0\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:13.761040 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:13.761006 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:14.221878 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:14.219845 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" podStartSLOduration=3.103037357 podStartE2EDuration="8.219825175s" totalImagesPullingTime="5.116787818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:13.706459586 +0000 UTC m=+41.675279152" watchObservedRunningTime="2026-10-30 21:55:14.219825175 +0000 UTC m=+42.188644743" Oct 30 21:55:14.221878 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:14.220150 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:55:14.231528 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:55:14.231492 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebc581_e471_4b4b_9162_b384a5d1fa43.slice/crio-07ecd1b1549586cfd379d4879167ecb9e6263e5b96c936cde8a8dbf03403ab23 WatchSource:0}: Error finding container 07ecd1b1549586cfd379d4879167ecb9e6263e5b96c936cde8a8dbf03403ab23: Status 404 returned error can't find the container with id 07ecd1b1549586cfd379d4879167ecb9e6263e5b96c936cde8a8dbf03403ab23 Oct 30 21:55:14.465084 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:14.465048 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"07ecd1b1549586cfd379d4879167ecb9e6263e5b96c936cde8a8dbf03403ab23"} Oct 30 21:55:16.474271 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:16.474200 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" exitCode=0 Oct 30 21:55:16.474669 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:16.474271 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} Oct 30 21:55:19.471870 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:19.471839 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-588b9668b8-xsbwg" Oct 30 21:55:21.499157 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499119 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} Oct 30 21:55:21.499543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499210 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} Oct 30 21:55:21.499543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499236 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} Oct 30 21:55:21.499543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} Oct 30 21:55:21.499543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499282 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} Oct 30 21:55:21.499543 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.499304 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerStarted","Data":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} Oct 30 21:55:21.674189 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:21.674127 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.814165472 podStartE2EDuration="8.674108776s" totalImagesPullingTime="5.840464832s" totalInitContainerRuntime="19.478472ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:21.67276529 +0000 UTC m=+49.641584853" watchObservedRunningTime="2026-10-30 21:55:21.674108776 +0000 UTC m=+49.642928341" Oct 30 21:55:23.619105 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:23.619062 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:23.761859 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:23.761827 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:26.638995 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:55:26.638957 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:55:27.101517 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:27.101478 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:27.150623 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:27.150591 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:27.534169 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:27.534098 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:55:48.639824 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:48.639761 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f44b89489-fwk5c" podUID="68f329eb-8a5a-445d-8db6-3203b29116f3" containerName="console" containerID="cri-o://68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368" gracePeriod=15 Oct 30 21:55:48.911383 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:48.911326 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:49.007533 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007508 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007645 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007565 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007645 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007625 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007718 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007653 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007718 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007697 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007788 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007727 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\" (UniqueName: \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007788 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007767 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\") pod \"68f329eb-8a5a-445d-8db6-3203b29116f3\" (UID: \"68f329eb-8a5a-445d-8db6-3203b29116f3\") " Oct 30 21:55:49.007914 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007883 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:49.008000 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.007982 4292 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68f329eb-8a5a-445d-8db6-3203b29116f3-tmp\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.008113 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.008090 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:49.008225 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.008182 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:49.008225 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.008194 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:49.009763 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.009738 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:49.009887 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.009863 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "kube-api-access-8cltw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:49.009943 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.009877 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert" pod "68f329eb-8a5a-445d-8db6-3203b29116f3" (UID: "68f329eb-8a5a-445d-8db6-3203b29116f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:49.108922 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108893 4292 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-console-config\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.108922 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108918 4292 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-oauth-serving-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.109074 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108928 4292 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-serving-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.109074 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108936 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cltw\" (UniqueName: \"kubernetes.io/projected/68f329eb-8a5a-445d-8db6-3203b29116f3-kube-api-access-8cltw\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.109074 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108946 4292 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f329eb-8a5a-445d-8db6-3203b29116f3-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.109074 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.108955 4292 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f329eb-8a5a-445d-8db6-3203b29116f3-console-oauth-config\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:49.599434 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.599400 4292 generic.go:356] "Generic (PLEG): container finished" podID="68f329eb-8a5a-445d-8db6-3203b29116f3" containerID="68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368" exitCode=0 Oct 30 21:55:49.599434 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.599435 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f44b89489-fwk5c" event={"ID":"68f329eb-8a5a-445d-8db6-3203b29116f3","Type":"ContainerDied","Data":"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368"} Oct 30 21:55:49.599742 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.599449 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f44b89489-fwk5c" Oct 30 21:55:49.599742 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.599468 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f44b89489-fwk5c" event={"ID":"68f329eb-8a5a-445d-8db6-3203b29116f3","Type":"ContainerDied","Data":"db3db6f89d2eb83d333ba7923e82d08e1774696a40cfc8686eba7f0088352783"} Oct 30 21:55:49.599742 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.599493 4292 scope.go:122] "RemoveContainer" containerID="68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368" Oct 30 21:55:49.608036 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.608020 4292 scope.go:122] "RemoveContainer" containerID="68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368" Oct 30 21:55:49.608298 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:55:49.608278 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368\": container with ID starting with 68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368 not found: ID does not exist" containerID="68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368" Oct 30 21:55:49.608362 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.608305 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368"} err="failed to get container status \"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368\": rpc error: code = NotFound desc = could not find container \"68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368\": container with ID starting with 68702cf2738a3a8d1192bb5d4b38f098ab6ba578bf5a89e41b5a258d32f0b368 not found: ID does not exist" Oct 30 21:55:49.767641 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.767612 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:49.819865 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:49.819829 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f44b89489-fwk5c"] Oct 30 21:55:50.143591 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:55:50.143543 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f329eb-8a5a-445d-8db6-3203b29116f3" path="/var/lib/kubelet/pods/68f329eb-8a5a-445d-8db6-3203b29116f3/volumes" Oct 30 21:55:56.638956 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:55:56.638920 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:56:03.353719 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.353681 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:03.354118 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354066 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="prometheus" containerID="cri-o://0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" gracePeriod=600 Oct 30 21:56:03.354185 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354100 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy" containerID="cri-o://b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" gracePeriod=600 Oct 30 21:56:03.354242 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354155 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="config-reloader" containerID="cri-o://6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" gracePeriod=600 Oct 30 21:56:03.354242 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354105 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="thanos-sidecar" containerID="cri-o://e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" gracePeriod=600 Oct 30 21:56:03.354344 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354196 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-web" containerID="cri-o://dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" gracePeriod=600 Oct 30 21:56:03.354344 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.354266 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" gracePeriod=600 Oct 30 21:56:03.609442 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.609387 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:03.651740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651711 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" exitCode=0 Oct 30 21:56:03.651740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651732 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" exitCode=0 Oct 30 21:56:03.651740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651741 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" exitCode=0 Oct 30 21:56:03.651740 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651746 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" exitCode=0 Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651751 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" exitCode=0 Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651757 4292 generic.go:356] "Generic (PLEG): container finished" podID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" exitCode=0 Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651780 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651830 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651837 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651868 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651891 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651919 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.651948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651928 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} Oct 30 21:56:03.652232 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.651964 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} Oct 30 21:56:03.652232 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.652003 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28ebc581-e471-4b4b-9162-b384a5d1fa43","Type":"ContainerDied","Data":"07ecd1b1549586cfd379d4879167ecb9e6263e5b96c936cde8a8dbf03403ab23"} Oct 30 21:56:03.659123 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.659098 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.666256 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.666234 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.672612 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.672594 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.678646 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.678629 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.685125 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.685109 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.691456 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.691441 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.700169 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700147 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.700410 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.700389 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.700491 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700423 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.700491 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700445 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.700702 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.700686 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.700770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700711 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.700770 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700730 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.700984 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.700969 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.701045 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.700989 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.701045 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701006 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.701219 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.701203 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.701286 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701224 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.701286 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701244 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.701468 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.701451 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.701509 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701475 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.701509 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701489 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.701744 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.701727 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.701813 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701751 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.701813 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.701772 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.702033 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:03.702016 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.702069 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702037 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.702069 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702050 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.702245 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702227 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.702304 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702246 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.702475 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702456 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.702519 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702476 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.702717 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702699 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.702793 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702718 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.702942 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702925 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.702985 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.702943 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.703165 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703147 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.703232 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703166 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.703373 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703356 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.703425 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703374 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.703566 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703548 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.703616 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703566 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.703760 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703743 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.703806 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703760 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.703937 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703922 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.703981 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.703939 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.704148 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704130 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.704192 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704148 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.704377 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704355 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.704377 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704374 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.704616 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704596 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.704674 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704617 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.704835 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704818 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.704835 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.704834 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.705035 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705019 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.705035 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705034 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.705250 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705235 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.705290 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705250 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.705441 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705423 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.705482 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705443 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.705685 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705670 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.705739 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705684 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.705870 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705856 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.705906 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.705870 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.706051 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706036 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.706091 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706050 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.706247 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706232 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.706290 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706246 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.706432 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706417 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.706472 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706431 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.706661 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706645 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.706661 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706660 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.706877 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706861 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.706877 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.706876 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.707074 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707059 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.707123 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707074 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.707274 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707252 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.707318 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707274 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.707469 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707453 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.707469 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707468 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.707686 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707671 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.707686 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707685 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.707873 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707858 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.707916 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.707873 4292 scope.go:122] "RemoveContainer" containerID="0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51" Oct 30 21:56:03.708056 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708040 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51"} err="failed to get container status \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": rpc error: code = NotFound desc = could not find container \"0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51\": container with ID starting with 0034195ca5c3799252b63b824315af62f84e13161ec37c52811429b153beef51 not found: ID does not exist" Oct 30 21:56:03.708097 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708056 4292 scope.go:122] "RemoveContainer" containerID="b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d" Oct 30 21:56:03.708228 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708213 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d"} err="failed to get container status \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": rpc error: code = NotFound desc = could not find container \"b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d\": container with ID starting with b6141743dc299555d867a6e2f8454c322f2488ffd85b7a3d14760e08ad04570d not found: ID does not exist" Oct 30 21:56:03.708268 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708227 4292 scope.go:122] "RemoveContainer" containerID="dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54" Oct 30 21:56:03.708392 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708376 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54"} err="failed to get container status \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": rpc error: code = NotFound desc = could not find container \"dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54\": container with ID starting with dc27ec27b36ea8fdaaff00739f955acf47c017c794cf2fca77296f9537c33c54 not found: ID does not exist" Oct 30 21:56:03.708392 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708390 4292 scope.go:122] "RemoveContainer" containerID="e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b" Oct 30 21:56:03.708557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708541 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b"} err="failed to get container status \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": rpc error: code = NotFound desc = could not find container \"e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b\": container with ID starting with e3ea2695acb72e7d3adc36bba3cae29c34593bf0c86ef9b86a26452042badb8b not found: ID does not exist" Oct 30 21:56:03.708557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708555 4292 scope.go:122] "RemoveContainer" containerID="6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756" Oct 30 21:56:03.708748 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708727 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756"} err="failed to get container status \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": rpc error: code = NotFound desc = could not find container \"6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756\": container with ID starting with 6b9362623941987a93d23af52bd8dd69e4142568e98119ab424c3215ef5c8756 not found: ID does not exist" Oct 30 21:56:03.708817 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708749 4292 scope.go:122] "RemoveContainer" containerID="0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3" Oct 30 21:56:03.708964 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708946 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3"} err="failed to get container status \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": rpc error: code = NotFound desc = could not find container \"0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3\": container with ID starting with 0830a644971dd3d8b021ca8f22000b71f3383a2ee6824cc2bb5e55d24d34f9c3 not found: ID does not exist" Oct 30 21:56:03.709029 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.708965 4292 scope.go:122] "RemoveContainer" containerID="05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f" Oct 30 21:56:03.709174 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.709158 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f"} err="failed to get container status \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": rpc error: code = NotFound desc = could not find container \"05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f\": container with ID starting with 05df0d1cad4dabbd5ef920b9e1fdf10f23149a40e21f566ebcd13b0b2f2cc83f not found: ID does not exist" Oct 30 21:56:03.725612 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725593 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.725696 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725621 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.725696 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725640 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.725696 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.725696 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725685 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.725897 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.725716 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726276 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726187 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726276 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726261 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726326 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726356 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.726408 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726374 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726415 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726459 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726398 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.726556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726506 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726556 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726538 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726824 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726569 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.726824 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726606 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.726824 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.726789 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.728892 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.728742 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.728892 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.728826 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "kube-api-access-8khfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:03.729845 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729481 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.729845 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729521 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.729845 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.729845 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729739 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.729845 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.730161 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729843 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.730161 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729941 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.730161 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729926 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.730161 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.729981 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\") pod \"28ebc581-e471-4b4b-9162-b384a5d1fa43\" (UID: \"28ebc581-e471-4b4b-9162-b384a5d1fa43\") " Oct 30 21:56:03.731054 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731025 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:03.731317 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731293 4292 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731391 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731324 4292 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731391 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731348 4292 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731391 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731364 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khfm\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-kube-api-access-8khfm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731391 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731381 4292 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731399 4292 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731419 4292 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-metrics-client-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731434 4292 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731451 4292 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731557 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731468 4292 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-grpc-tls\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.731795 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731729 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:03.731795 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.731742 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:03.733136 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.733063 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:03.734396 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.734372 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.735217 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.735194 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.735309 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.735216 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.735309 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.735228 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.735309 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.735246 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.742800 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.742781 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config" pod "28ebc581-e471-4b4b-9162-b384a5d1fa43" (UID: "28ebc581-e471-4b4b-9162-b384a5d1fa43"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:03.832474 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832445 4292 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28ebc581-e471-4b4b-9162-b384a5d1fa43-tls-assets\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832474 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832470 4292 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-metrics-client-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832474 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832480 4292 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832489 4292 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832499 4292 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-config\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832507 4292 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-secret-kube-rbac-proxy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832515 4292 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832524 4292 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-prometheus-k8s-db\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832532 4292 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28ebc581-e471-4b4b-9162-b384a5d1fa43-config-out\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.832697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:03.832540 4292 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28ebc581-e471-4b4b-9162-b384a5d1fa43-web-config\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:04.076321 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.076294 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:04.082184 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.082163 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:04.118984 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.118959 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:04.119269 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119255 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="init-config-reloader" Oct 30 21:56:04.119316 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119271 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="init-config-reloader" Oct 30 21:56:04.119316 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119284 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-thanos" Oct 30 21:56:04.119316 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119289 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-thanos" Oct 30 21:56:04.119316 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119309 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="config-reloader" Oct 30 21:56:04.119316 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119317 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="config-reloader" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119325 4292 cpu_manager.go:414] "containerMap: removing container" podUID="68f329eb-8a5a-445d-8db6-3203b29116f3" containerName="console" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119330 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f329eb-8a5a-445d-8db6-3203b29116f3" containerName="console" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119337 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="thanos-sidecar" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119343 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="thanos-sidecar" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119352 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="prometheus" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119357 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="prometheus" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119367 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-web" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119371 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-web" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119377 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119383 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119426 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="config-reloader" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119435 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f329eb-8a5a-445d-8db6-3203b29116f3" containerName="console" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119441 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="thanos-sidecar" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119447 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="prometheus" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119453 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-web" Oct 30 21:56:04.119455 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119460 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy" Oct 30 21:56:04.119951 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.119470 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" containerName="kube-rbac-proxy-thanos" Oct 30 21:56:04.123003 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.122987 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.143562 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.143538 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ebc581-e471-4b4b-9162-b384a5d1fa43" path="/var/lib/kubelet/pods/28ebc581-e471-4b4b-9162-b384a5d1fa43/volumes" Oct 30 21:56:04.177384 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177358 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:56:04.177483 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177396 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:56:04.177872 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177848 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:56:04.177982 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177869 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:56:04.177982 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177893 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:56:04.177982 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.177896 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:56:04.178160 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178104 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:56:04.178160 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178139 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jcc4t\"" Oct 30 21:56:04.178263 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178166 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:56:04.178263 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178245 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:56:04.178374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178305 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:56:04.178374 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178362 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:56:04.178478 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178407 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:56:04.178478 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178415 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:56:04.178478 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178463 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9v1fbagc363of\"" Oct 30 21:56:04.178777 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.178763 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:56:04.229076 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.229026 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:56:04.235344 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235324 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235409 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235356 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235409 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235377 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235471 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235449 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-web-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235537 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235525 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235590 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235640 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235611 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235668 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235657 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235697 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235674 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235731 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235694 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235731 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235711 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235793 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235733 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235825 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235791 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8whs\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-kube-api-access-q8whs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235856 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235824 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235856 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235849 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235915 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235866 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235948 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235986 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235957 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.235986 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235976 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-config-out\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.236046 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.235992 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.249540 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.249514 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:04.336813 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336789 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336818 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-web-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336837 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336856 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336872 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336891 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.336930 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336908 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336933 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336950 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336969 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.336986 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8whs\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-kube-api-access-q8whs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337006 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337040 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337061 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337085 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337104 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-config-out\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337119 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337149 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337203 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337174 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.337850 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.337616 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.338380 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.338321 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.338733 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.338692 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.339176 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.338871 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.340271 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-web-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.340329 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.340351 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.340751 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.340810 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341265 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.341225 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.341708 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.341394 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342361 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.341803 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342361 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.342296 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-config\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342494 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.342374 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342494 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.342068 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342656 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.342616 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c76233e0-8e09-4f9e-8e9e-963f8462516e-config-out\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.342803 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.342757 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c76233e0-8e09-4f9e-8e9e-963f8462516e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.344865 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.344841 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.345929 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.345910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c76233e0-8e09-4f9e-8e9e-963f8462516e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.400888 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.400856 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8whs\" (UniqueName: \"kubernetes.io/projected/c76233e0-8e09-4f9e-8e9e-963f8462516e-kube-api-access-q8whs\") pod \"prometheus-k8s-0\" (UID: \"c76233e0-8e09-4f9e-8e9e-963f8462516e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.440977 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.440949 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:04.872170 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:04.872143 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:56:04.874450 ip-10-0-142-210 kubenswrapper[4292]: W1030 21:56:04.874414 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76233e0_8e09_4f9e_8e9e_963f8462516e.slice/crio-9aa74e1bafdad41f12f980d39330209d7b61af3096bc34d63bf8e918e2957e2b WatchSource:0}: Error finding container 9aa74e1bafdad41f12f980d39330209d7b61af3096bc34d63bf8e918e2957e2b: Status 404 returned error can't find the container with id 9aa74e1bafdad41f12f980d39330209d7b61af3096bc34d63bf8e918e2957e2b Oct 30 21:56:05.660952 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:05.660920 4292 generic.go:356] "Generic (PLEG): container finished" podID="c76233e0-8e09-4f9e-8e9e-963f8462516e" containerID="4d42d57b4634b33a06b0ecb7ee99cc0e41f85f1fb871ab3d5b6110bb633b87c6" exitCode=0 Oct 30 21:56:05.661331 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:05.660961 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerDied","Data":"4d42d57b4634b33a06b0ecb7ee99cc0e41f85f1fb871ab3d5b6110bb633b87c6"} Oct 30 21:56:05.661331 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:05.661011 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"9aa74e1bafdad41f12f980d39330209d7b61af3096bc34d63bf8e918e2957e2b"} Oct 30 21:56:06.668378 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668341 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"39d343daed9e07f5b5615303d780910b1b4bea0cbea5d54077409a94958877f3"} Oct 30 21:56:06.668780 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668410 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"8c1d8e0603f2a4b2ccdabfb68c7be08c696236d289274fe00a4da2f4cacbeb3f"} Oct 30 21:56:06.668780 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668436 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"d6fe37e0d2f5cf01e765f81b1a1546a6a02e2a9b117520b984d7bb1665c49de2"} Oct 30 21:56:06.668780 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668459 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"179f29b9ebcb97d6bab148aeda570a2e1850589298041bceecb70ada2eb4495c"} Oct 30 21:56:06.668780 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668488 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"26197428789d32f9c6ddbcb6660be6dae78b7c66750bd24d4104ae0906a41dd8"} Oct 30 21:56:06.668780 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.668517 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c76233e0-8e09-4f9e-8e9e-963f8462516e","Type":"ContainerStarted","Data":"1d64c32c7979e1895c8e7c8055a91dc18f7b1d7f44751faaa3059ea06198a3d0"} Oct 30 21:56:06.806385 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:06.806336 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.791113638 podStartE2EDuration="2.806322508s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.20887ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:06.805453222 +0000 UTC m=+94.774272787" watchObservedRunningTime="2026-10-30 21:56:06.806322508 +0000 UTC m=+94.775142071" Oct 30 21:56:09.441915 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:56:09.441872 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:56:26.638313 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:26.638254 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:56:56.638269 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:56:56.638188 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:57:04.441471 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:57:04.441433 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:57:04.456619 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:57:04.456594 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:57:04.885667 ip-10-0-142-210 kubenswrapper[4292]: I1030 21:57:04.885638 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:57:26.638863 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:57:26.638819 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:57:56.637966 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:57:56.637925 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:58:26.638157 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:58:26.638121 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:58:56.637287 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:58:56.637241 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:59:26.638236 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:59:26.638195 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 21:59:56.638280 ip-10-0-142-210 kubenswrapper[4292]: E1030 21:59:56.638203 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:00:26.638705 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:00:26.638668 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:00:38.585158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.585122 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7"] Oct 30 22:00:38.587634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.587619 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.642859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.642833 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.693161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.693140 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.693381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.693363 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-2v7gs\"" Oct 30 22:00:38.694431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.694414 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7"] Oct 30 22:00:38.699724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.699707 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlrf\" (UniqueName: \"kubernetes.io/projected/1cde1368-4b51-4163-8e13-c7f6b1c845ed-kube-api-access-5hlrf\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.699791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.699744 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cde1368-4b51-4163-8e13-c7f6b1c845ed-tmp\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.800426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.800398 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cde1368-4b51-4163-8e13-c7f6b1c845ed-tmp\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.800541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.800455 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlrf\" (UniqueName: \"kubernetes.io/projected/1cde1368-4b51-4163-8e13-c7f6b1c845ed-kube-api-access-5hlrf\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.800758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.800742 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cde1368-4b51-4163-8e13-c7f6b1c845ed-tmp\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.863618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.863554 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlrf\" (UniqueName: \"kubernetes.io/projected/1cde1368-4b51-4163-8e13-c7f6b1c845ed-kube-api-access-5hlrf\") pod \"jobset-operator-674cd59dc5-bfkj7\" (UID: \"1cde1368-4b51-4163-8e13-c7f6b1c845ed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:38.905360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:38.905341 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" Oct 30 22:00:39.278141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:39.278078 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7"] Oct 30 22:00:39.280789 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:00:39.280764 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cde1368_4b51_4163_8e13_c7f6b1c845ed.slice/crio-39211d28ae504e4a75b68005402f19b65416595437101525469424060f57cf60 WatchSource:0}: Error finding container 39211d28ae504e4a75b68005402f19b65416595437101525469424060f57cf60: Status 404 returned error can't find the container with id 39211d28ae504e4a75b68005402f19b65416595437101525469424060f57cf60 Oct 30 22:00:39.282300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:39.282282 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:00:39.561934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:39.561903 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" event={"ID":"1cde1368-4b51-4163-8e13-c7f6b1c845ed","Type":"ContainerStarted","Data":"39211d28ae504e4a75b68005402f19b65416595437101525469424060f57cf60"} Oct 30 22:00:46.082746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.081175 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn"] Oct 30 22:00:46.086000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.084802 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.195245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.195163 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 22:00:46.195245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.195167 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 22:00:46.195245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.195207 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 22:00:46.195245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.195232 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9rb8f\"" Oct 30 22:00:46.200222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.200199 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 22:00:46.200345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.200267 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 22:00:46.255381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255352 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkvf\" (UniqueName: \"kubernetes.io/projected/47cad9db-3197-421a-b5c4-56f1500dde05-kube-api-access-jnkvf\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.255506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255394 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-tmp\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.255506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255413 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.255635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255613 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.255675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255648 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-manifests\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.255708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.255682 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.336381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.336352 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn"] Oct 30 22:00:46.356183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356158 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356184 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-manifests\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356205 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356232 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkvf\" (UniqueName: \"kubernetes.io/projected/47cad9db-3197-421a-b5c4-56f1500dde05-kube-api-access-jnkvf\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356340 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-tmp\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356371 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.356855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.356830 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-manifests\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.358874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.358849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.358965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.358896 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47cad9db-3197-421a-b5c4-56f1500dde05-tmp\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.359096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.359076 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.359134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.359114 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47cad9db-3197-421a-b5c4-56f1500dde05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.421623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.421598 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkvf\" (UniqueName: \"kubernetes.io/projected/47cad9db-3197-421a-b5c4-56f1500dde05-kube-api-access-jnkvf\") pod \"opendatahub-operator-controller-manager-f7fbfb566-v79cn\" (UID: \"47cad9db-3197-421a-b5c4-56f1500dde05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.589819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.589792 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" event={"ID":"1cde1368-4b51-4163-8e13-c7f6b1c845ed","Type":"ContainerStarted","Data":"694f5e206dc8ea3a3862507eb8e61fe978c629f4f499b6c37596587f6f260fa9"} Oct 30 22:00:46.704722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.704699 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:46.710385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:46.710349 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bfkj7" podStartSLOduration=2.112013134 podStartE2EDuration="8.710338389s" totalImagesPullingTime="6.598325255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:46.709553068 +0000 UTC m=+374.678372632" watchObservedRunningTime="2026-10-30 22:00:46.710338389 +0000 UTC m=+374.679157952" Oct 30 22:00:47.249424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:47.249389 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn"] Oct 30 22:00:47.250439 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:00:47.250412 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cad9db_3197_421a_b5c4_56f1500dde05.slice/crio-8c79a8e95783e62e35c424bd035d075f0f2ead291827e5c7f0a125fd661d70cf WatchSource:0}: Error finding container 8c79a8e95783e62e35c424bd035d075f0f2ead291827e5c7f0a125fd661d70cf: Status 404 returned error can't find the container with id 8c79a8e95783e62e35c424bd035d075f0f2ead291827e5c7f0a125fd661d70cf Oct 30 22:00:47.594552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:47.594519 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" event={"ID":"47cad9db-3197-421a-b5c4-56f1500dde05","Type":"ContainerStarted","Data":"8c79a8e95783e62e35c424bd035d075f0f2ead291827e5c7f0a125fd661d70cf"} Oct 30 22:00:51.611219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:51.611185 4292 generic.go:356] "Generic (PLEG): container finished" podID="47cad9db-3197-421a-b5c4-56f1500dde05" containerID="b1258f4275eaf5d209c01a5b6ead24562c0668ce43f05f0bf2eee9326ea9d7dd" exitCode=0 Oct 30 22:00:51.611622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:51.611269 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" event={"ID":"47cad9db-3197-421a-b5c4-56f1500dde05","Type":"ContainerDied","Data":"b1258f4275eaf5d209c01a5b6ead24562c0668ce43f05f0bf2eee9326ea9d7dd"} Oct 30 22:00:52.616898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:52.616864 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" event={"ID":"47cad9db-3197-421a-b5c4-56f1500dde05","Type":"ContainerStarted","Data":"3376b68717cbea43e9713215e0e19f6064f021f0efa6c0c852c189cc45a7ea1f"} Oct 30 22:00:52.617253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:52.617012 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:00:52.866919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:00:52.866872 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" podStartSLOduration=3.126835572 podStartE2EDuration="7.866858396s" totalImagesPullingTime="4.321141524s" totalInitContainerRuntime="418.8813ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:52.866244952 +0000 UTC m=+380.835064516" watchObservedRunningTime="2026-10-30 22:00:52.866858396 +0000 UTC m=+380.835677959" Oct 30 22:00:56.637937 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:00:56.637901 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:01:03.045410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.045376 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql"] Oct 30 22:01:03.053118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.053092 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.085190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.085163 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xv8\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-kube-api-access-x6xv8\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.085295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.085201 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.119969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.119947 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql"] Oct 30 22:01:03.168864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.168835 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 22:01:03.168981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.168888 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-krvd2\"" Oct 30 22:01:03.168981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.168899 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 22:01:03.186360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.186337 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.186448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.186432 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xv8\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-kube-api-access-x6xv8\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.249630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.249610 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xv8\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-kube-api-access-x6xv8\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.254554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.254530 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdab93d-64bb-41e6-8be7-f7bae6825a5c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xhqql\" (UID: \"dcdab93d-64bb-41e6-8be7-f7bae6825a5c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.363808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.363741 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:03.623296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.623232 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7fbfb566-v79cn" Oct 30 22:01:03.735139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:03.735087 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql"] Oct 30 22:01:03.737164 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:01:03.737130 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcdab93d_64bb_41e6_8be7_f7bae6825a5c.slice/crio-885d12ba8ab33144866438ccb6038113ec1e1daa870823796c92a1bf8c47f2bb WatchSource:0}: Error finding container 885d12ba8ab33144866438ccb6038113ec1e1daa870823796c92a1bf8c47f2bb: Status 404 returned error can't find the container with id 885d12ba8ab33144866438ccb6038113ec1e1daa870823796c92a1bf8c47f2bb Oct 30 22:01:04.661653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:04.661616 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" event={"ID":"dcdab93d-64bb-41e6-8be7-f7bae6825a5c","Type":"ContainerStarted","Data":"885d12ba8ab33144866438ccb6038113ec1e1daa870823796c92a1bf8c47f2bb"} Oct 30 22:01:07.170229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.170191 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4h677"] Oct 30 22:01:07.191459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.191435 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.218176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.218147 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-bound-sa-token\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.218331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.218229 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdzl\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-kube-api-access-cxdzl\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.256663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.256639 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4h677"] Oct 30 22:01:07.305430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.305397 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nc2xp\"" Oct 30 22:01:07.318786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.318754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-bound-sa-token\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.318907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.318833 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdzl\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-kube-api-access-cxdzl\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.386568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.386543 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-bound-sa-token\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.386735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.386648 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdzl\" (UniqueName: \"kubernetes.io/projected/b9338dc5-eeb9-4c2b-9fa8-e1495aec6192-kube-api-access-cxdzl\") pod \"cert-manager-597b95cb84-4h677\" (UID: \"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192\") " pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.514422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.514339 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4h677" Oct 30 22:01:07.916976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:07.916951 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4h677"] Oct 30 22:01:07.918045 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:01:07.918020 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9338dc5_eeb9_4c2b_9fa8_e1495aec6192.slice/crio-55a372368ac2ba24721c851ced881dada121458960f8db552305207e8ed29185 WatchSource:0}: Error finding container 55a372368ac2ba24721c851ced881dada121458960f8db552305207e8ed29185: Status 404 returned error can't find the container with id 55a372368ac2ba24721c851ced881dada121458960f8db552305207e8ed29185 Oct 30 22:01:08.681128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.681088 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" event={"ID":"dcdab93d-64bb-41e6-8be7-f7bae6825a5c","Type":"ContainerStarted","Data":"128640548cbfcaebb084656ecb1bc6e87b1c9e077b64d64299e7bf40a67435f9"} Oct 30 22:01:08.681557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.681262 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:08.682667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.682642 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4h677" event={"ID":"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192","Type":"ContainerStarted","Data":"49abbe21d9d582661ed2b4aa751211da58a57da1f29ef234a8ace668bffb22b6"} Oct 30 22:01:08.682757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.682675 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4h677" event={"ID":"b9338dc5-eeb9-4c2b-9fa8-e1495aec6192","Type":"ContainerStarted","Data":"55a372368ac2ba24721c851ced881dada121458960f8db552305207e8ed29185"} Oct 30 22:01:08.801619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.801559 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" podStartSLOduration=2.673594917 podStartE2EDuration="6.801547355s" totalImagesPullingTime="4.127952438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.801261497 +0000 UTC m=+396.770081062" watchObservedRunningTime="2026-10-30 22:01:08.801547355 +0000 UTC m=+396.770366913" Oct 30 22:01:08.937795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:08.937705 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4h677" podStartSLOduration=1.937693152 podStartE2EDuration="1.937693152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.936959359 +0000 UTC m=+396.905778937" watchObservedRunningTime="2026-10-30 22:01:08.937693152 +0000 UTC m=+396.906512716" Oct 30 22:01:14.688924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:14.688897 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" Oct 30 22:01:18.194692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.194661 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9"] Oct 30 22:01:18.197776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.197760 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.252337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.252311 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 22:01:18.252447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.252367 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 22:01:18.302028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.302001 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 22:01:18.302142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.302042 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 22:01:18.302142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.302092 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m79jn\"" Oct 30 22:01:18.303601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.303584 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9"] Oct 30 22:01:18.306486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.306468 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kkm\" (UniqueName: \"kubernetes.io/projected/669ff4ad-5e46-4f46-a21b-6b6e194643a2-kube-api-access-x2kkm\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.306532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.306508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669ff4ad-5e46-4f46-a21b-6b6e194643a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.306532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.306526 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669ff4ad-5e46-4f46-a21b-6b6e194643a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.407418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.407394 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kkm\" (UniqueName: \"kubernetes.io/projected/669ff4ad-5e46-4f46-a21b-6b6e194643a2-kube-api-access-x2kkm\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.407521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.407436 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669ff4ad-5e46-4f46-a21b-6b6e194643a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.407521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.407460 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669ff4ad-5e46-4f46-a21b-6b6e194643a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.408125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.408105 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669ff4ad-5e46-4f46-a21b-6b6e194643a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.409840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.409818 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669ff4ad-5e46-4f46-a21b-6b6e194643a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.477539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.477485 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kkm\" (UniqueName: \"kubernetes.io/projected/669ff4ad-5e46-4f46-a21b-6b6e194643a2-kube-api-access-x2kkm\") pod \"kuadrant-console-plugin-66c4fb575d-t48f9\" (UID: \"669ff4ad-5e46-4f46-a21b-6b6e194643a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.509251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.509228 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" Oct 30 22:01:18.880030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:18.880007 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9"] Oct 30 22:01:18.882290 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:01:18.882261 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod669ff4ad_5e46_4f46_a21b_6b6e194643a2.slice/crio-251ff07cda59f5329fefd704f0892a02afebd0a17d327e337e3a980382dadb63 WatchSource:0}: Error finding container 251ff07cda59f5329fefd704f0892a02afebd0a17d327e337e3a980382dadb63: Status 404 returned error can't find the container with id 251ff07cda59f5329fefd704f0892a02afebd0a17d327e337e3a980382dadb63 Oct 30 22:01:19.726550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:19.726512 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" event={"ID":"669ff4ad-5e46-4f46-a21b-6b6e194643a2","Type":"ContainerStarted","Data":"251ff07cda59f5329fefd704f0892a02afebd0a17d327e337e3a980382dadb63"} Oct 30 22:01:24.751704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:24.751672 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" event={"ID":"669ff4ad-5e46-4f46-a21b-6b6e194643a2","Type":"ContainerStarted","Data":"934ba4a653b162af6372d70d069712a27a0c14f89c710287d013398f2990650e"} Oct 30 22:01:24.871345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:24.871297 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t48f9" podStartSLOduration=1.121906779 podStartE2EDuration="6.871282167s" totalImagesPullingTime="5.749375388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:24.87116135 +0000 UTC m=+412.839980917" watchObservedRunningTime="2026-10-30 22:01:24.871282167 +0000 UTC m=+412.840101731" Oct 30 22:01:25.552469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.552434 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-679b756675-b9mf7"] Oct 30 22:01:25.555851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.555834 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.613253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.613227 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dkbh2\"" Oct 30 22:01:25.663776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.663742 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 22:01:25.663911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.663792 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 22:01:25.665096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.665076 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-679b756675-b9mf7"] Oct 30 22:01:25.668465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.668443 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e518fe02-f319-476c-a217-0a81262e620c-tmp\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.668566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.668505 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e518fe02-f319-476c-a217-0a81262e620c-tls-certs\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.668629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.668607 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bxg\" (UniqueName: \"kubernetes.io/projected/e518fe02-f319-476c-a217-0a81262e620c-kube-api-access-m6bxg\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.713495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.713478 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 22:01:25.713672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.713658 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 22:01:25.769374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.769352 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e518fe02-f319-476c-a217-0a81262e620c-tls-certs\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.769682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.769381 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bxg\" (UniqueName: \"kubernetes.io/projected/e518fe02-f319-476c-a217-0a81262e620c-kube-api-access-m6bxg\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.769682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.769440 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e518fe02-f319-476c-a217-0a81262e620c-tmp\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.771670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.771644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e518fe02-f319-476c-a217-0a81262e620c-tmp\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.771790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.771774 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e518fe02-f319-476c-a217-0a81262e620c-tls-certs\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.832766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.832709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bxg\" (UniqueName: \"kubernetes.io/projected/e518fe02-f319-476c-a217-0a81262e620c-kube-api-access-m6bxg\") pod \"kube-auth-proxy-679b756675-b9mf7\" (UID: \"e518fe02-f319-476c-a217-0a81262e620c\") " pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:25.867506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:25.867484 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" Oct 30 22:01:26.329190 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:01:26.329160 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode518fe02_f319_476c_a217_0a81262e620c.slice/crio-ce16ffddaf29d5d32f8ebb591d7d1eeb0d945a6822edaa1bd1ab001686c2487a WatchSource:0}: Error finding container ce16ffddaf29d5d32f8ebb591d7d1eeb0d945a6822edaa1bd1ab001686c2487a: Status 404 returned error can't find the container with id ce16ffddaf29d5d32f8ebb591d7d1eeb0d945a6822edaa1bd1ab001686c2487a Oct 30 22:01:26.329854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:26.329771 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-679b756675-b9mf7"] Oct 30 22:01:26.638129 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:01:26.638048 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:01:26.760355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:26.760323 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" event={"ID":"e518fe02-f319-476c-a217-0a81262e620c","Type":"ContainerStarted","Data":"ce16ffddaf29d5d32f8ebb591d7d1eeb0d945a6822edaa1bd1ab001686c2487a"} Oct 30 22:01:29.775051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:29.775014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" event={"ID":"e518fe02-f319-476c-a217-0a81262e620c","Type":"ContainerStarted","Data":"2a72deb4c3624a633b0b366195b6bd68a5ed6785c0fc13cb5d3c72d2c88507dc"} Oct 30 22:01:29.899689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:29.899642 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-679b756675-b9mf7" podStartSLOduration=1.960237598 podStartE2EDuration="4.899629581s" totalImagesPullingTime="2.939391983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:29.89879322 +0000 UTC m=+417.867612784" watchObservedRunningTime="2026-10-30 22:01:29.899629581 +0000 UTC m=+417.868449145" Oct 30 22:01:37.588590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.588534 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9"] Oct 30 22:01:37.593277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.593255 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.665492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.665463 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmmv\" (UniqueName: \"kubernetes.io/projected/14d1e346-f6cd-408b-bf6b-c7ac88038944-kube-api-access-qgmmv\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.665675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.665497 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-cert\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.665675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.665539 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-metrics-certs\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.665675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.665640 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14d1e346-f6cd-408b-bf6b-c7ac88038944-manager-config\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.697260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.697236 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 22:01:37.697598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.697569 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 22:01:37.697658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.697612 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6pzc8\"" Oct 30 22:01:37.697658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.697648 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 22:01:37.699260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.699241 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9"] Oct 30 22:01:37.766062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.766038 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmmv\" (UniqueName: \"kubernetes.io/projected/14d1e346-f6cd-408b-bf6b-c7ac88038944-kube-api-access-qgmmv\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.766167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.766066 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-cert\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.766167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.766108 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-metrics-certs\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.766167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.766134 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14d1e346-f6cd-408b-bf6b-c7ac88038944-manager-config\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.766832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.766811 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14d1e346-f6cd-408b-bf6b-c7ac88038944-manager-config\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.768490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.768474 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-cert\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.768559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.768541 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d1e346-f6cd-408b-bf6b-c7ac88038944-metrics-certs\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.831272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.831249 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmmv\" (UniqueName: \"kubernetes.io/projected/14d1e346-f6cd-408b-bf6b-c7ac88038944-kube-api-access-qgmmv\") pod \"jobset-controller-manager-5bb496bc44-q6bj9\" (UID: \"14d1e346-f6cd-408b-bf6b-c7ac88038944\") " pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:37.905235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:37.905186 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:38.289250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:38.289225 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9"] Oct 30 22:01:38.291519 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:01:38.291498 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d1e346_f6cd_408b_bf6b_c7ac88038944.slice/crio-cc65fb2784328594948b833e971045f2288cde2f5a5c8a30c922c23d0f50dd7a WatchSource:0}: Error finding container cc65fb2784328594948b833e971045f2288cde2f5a5c8a30c922c23d0f50dd7a: Status 404 returned error can't find the container with id cc65fb2784328594948b833e971045f2288cde2f5a5c8a30c922c23d0f50dd7a Oct 30 22:01:38.809923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:38.809890 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" event={"ID":"14d1e346-f6cd-408b-bf6b-c7ac88038944","Type":"ContainerStarted","Data":"cc65fb2784328594948b833e971045f2288cde2f5a5c8a30c922c23d0f50dd7a"} Oct 30 22:01:42.829284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:42.829242 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" event={"ID":"14d1e346-f6cd-408b-bf6b-c7ac88038944","Type":"ContainerStarted","Data":"4f2e1cb34a320c8f21121b611419cf977ffab7ce9b2e968e21fcac3c2141d745"} Oct 30 22:01:42.829685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:42.829386 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:42.949711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:42.949667 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" podStartSLOduration=2.162421418 podStartE2EDuration="5.949655154s" totalImagesPullingTime="3.787233736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:42.949194614 +0000 UTC m=+430.918014172" watchObservedRunningTime="2026-10-30 22:01:42.949655154 +0000 UTC m=+430.918474718" Oct 30 22:01:53.838188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:01:53.838151 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5bb496bc44-q6bj9" Oct 30 22:01:56.638555 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:01:56.638520 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:02:26.637903 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:02:26.637863 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:02:48.657050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.656976 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7vm5c"] Oct 30 22:02:48.659197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.659175 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.714426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.714399 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 22:02:48.724995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.724969 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dfg\" (UniqueName: \"kubernetes.io/projected/516f0f5e-adba-4e83-b968-9de5e9066b0a-kube-api-access-h5dfg\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.725128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.725027 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516f0f5e-adba-4e83-b968-9de5e9066b0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.765182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.765152 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-wpnwb\"" Oct 30 22:02:48.765182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.765152 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 22:02:48.765325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.765158 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 22:02:48.766539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.766520 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7vm5c"] Oct 30 22:02:48.825799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.825773 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dfg\" (UniqueName: \"kubernetes.io/projected/516f0f5e-adba-4e83-b968-9de5e9066b0a-kube-api-access-h5dfg\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.825921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.825817 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516f0f5e-adba-4e83-b968-9de5e9066b0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.828764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.828736 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516f0f5e-adba-4e83-b968-9de5e9066b0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.890660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.890633 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dfg\" (UniqueName: \"kubernetes.io/projected/516f0f5e-adba-4e83-b968-9de5e9066b0a-kube-api-access-h5dfg\") pod \"model-serving-api-75bd89fdfc-7vm5c\" (UID: \"516f0f5e-adba-4e83-b968-9de5e9066b0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:48.975390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:48.975328 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:49.343453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:49.343431 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7vm5c"] Oct 30 22:02:49.345772 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:02:49.345744 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516f0f5e_adba_4e83_b968_9de5e9066b0a.slice/crio-b6a513257df3231a12fbe8902bfd9668022d319d059453db5916a16d4804a796 WatchSource:0}: Error finding container b6a513257df3231a12fbe8902bfd9668022d319d059453db5916a16d4804a796: Status 404 returned error can't find the container with id b6a513257df3231a12fbe8902bfd9668022d319d059453db5916a16d4804a796 Oct 30 22:02:50.096033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:50.096002 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" event={"ID":"516f0f5e-adba-4e83-b968-9de5e9066b0a","Type":"ContainerStarted","Data":"b6a513257df3231a12fbe8902bfd9668022d319d059453db5916a16d4804a796"} Oct 30 22:02:52.107237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:52.107201 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" event={"ID":"516f0f5e-adba-4e83-b968-9de5e9066b0a","Type":"ContainerStarted","Data":"f80072c40eb43d8408fa6f976ab0f42574f7a3ca44b9b5a9fc46080b6d032480"} Oct 30 22:02:52.107665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:52.107286 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:02:52.228483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:02:52.228438 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" podStartSLOduration=2.2087340109999998 podStartE2EDuration="4.228426249s" totalImagesPullingTime="2.019692238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:52.226866238 +0000 UTC m=+500.195685801" watchObservedRunningTime="2026-10-30 22:02:52.228426249 +0000 UTC m=+500.197245813" Oct 30 22:02:56.638301 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:02:56.638266 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:03:03.114864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:03:03.114836 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7vm5c" Oct 30 22:03:26.638404 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:03:26.638365 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:03:56.638318 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:03:56.638213 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:04:26.637678 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:04:26.637641 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:04:56.638320 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:04:56.638280 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:05:01.509060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.509020 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k"] Oct 30 22:05:01.511204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.511187 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.565878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.565848 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 22:05:01.615469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.615433 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 22:05:01.615469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.615450 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rdd77\"" Oct 30 22:05:01.615469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.615451 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 22:05:01.617244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.617221 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k"] Oct 30 22:05:01.628074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628049 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.628169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628080 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.628169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628100 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3a411-27a4-4f18-a085-593710bff7c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.628234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628210 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.628298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628282 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.628332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.628316 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579r9\" (UniqueName: \"kubernetes.io/projected/a3c3a411-27a4-4f18-a085-593710bff7c5-kube-api-access-579r9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728705 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728761 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728789 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579r9\" (UniqueName: \"kubernetes.io/projected/a3c3a411-27a4-4f18-a085-593710bff7c5-kube-api-access-579r9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728817 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728837 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.728995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.728859 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3a411-27a4-4f18-a085-593710bff7c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.729187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.729163 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.729233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.729210 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.729268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.729248 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.731181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.731157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3a411-27a4-4f18-a085-593710bff7c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.731544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.731523 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3a411-27a4-4f18-a085-593710bff7c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.793462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.793390 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579r9\" (UniqueName: \"kubernetes.io/projected/a3c3a411-27a4-4f18-a085-593710bff7c5-kube-api-access-579r9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2229k\" (UID: \"a3c3a411-27a4-4f18-a085-593710bff7c5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:01.822327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:01.822290 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:02.199862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:02.199833 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k"] Oct 30 22:05:02.202389 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:02.202362 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c3a411_27a4_4f18_a085_593710bff7c5.slice/crio-6492af3746e3f90d420d2c81178c4c6ab576e12d2fdf5abf746d416a9454e1fc WatchSource:0}: Error finding container 6492af3746e3f90d420d2c81178c4c6ab576e12d2fdf5abf746d416a9454e1fc: Status 404 returned error can't find the container with id 6492af3746e3f90d420d2c81178c4c6ab576e12d2fdf5abf746d416a9454e1fc Oct 30 22:05:02.623426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:02.623388 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" event={"ID":"a3c3a411-27a4-4f18-a085-593710bff7c5","Type":"ContainerStarted","Data":"6492af3746e3f90d420d2c81178c4c6ab576e12d2fdf5abf746d416a9454e1fc"} Oct 30 22:05:04.633487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:04.633446 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" event={"ID":"a3c3a411-27a4-4f18-a085-593710bff7c5","Type":"ContainerStarted","Data":"39ae719ac8649834a0ae4c0f5e70f303ad777dbb91c97b5abb10bdec0309d838"} Oct 30 22:05:04.822652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:04.822628 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:11.784535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.784484 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" podStartSLOduration=8.575545511 podStartE2EDuration="10.784470486s" totalImagesPullingTime="2.208924975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:04.771483155 +0000 UTC m=+632.740302718" watchObservedRunningTime="2026-10-30 22:05:11.784470486 +0000 UTC m=+639.753290050" Oct 30 22:05:11.785012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.784996 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf"] Oct 30 22:05:11.787231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.787215 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.822973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.822950 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:11.835546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.835524 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:11.843304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.843286 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 22:05:11.893183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.893156 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf"] Oct 30 22:05:11.924054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924028 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pn7\" (UniqueName: \"kubernetes.io/projected/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-kube-api-access-d8pn7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.924150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924062 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.924150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.924214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924174 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.924261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924247 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:11.924296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:11.924272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025447 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pn7\" (UniqueName: \"kubernetes.io/projected/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-kube-api-access-d8pn7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025481 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025500 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025537 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025596 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025625 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025879 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.025964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025943 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.026010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.025995 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.027798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.027775 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.028061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.028045 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.089305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.089276 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pn7\" (UniqueName: \"kubernetes.io/projected/bc811bd0-2ab2-46b1-8d32-95cab341dcd7-kube-api-access-d8pn7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-hm5xf\" (UID: \"bc811bd0-2ab2-46b1-8d32-95cab341dcd7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.097964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.097944 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:12.493557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.493531 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf"] Oct 30 22:05:12.495352 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:12.495319 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc811bd0_2ab2_46b1_8d32_95cab341dcd7.slice/crio-c0137d73a8947a5c997a378274c2b27da4c046b460903b57b59363f7b4ae733b WatchSource:0}: Error finding container c0137d73a8947a5c997a378274c2b27da4c046b460903b57b59363f7b4ae733b: Status 404 returned error can't find the container with id c0137d73a8947a5c997a378274c2b27da4c046b460903b57b59363f7b4ae733b Oct 30 22:05:12.668304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.668266 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" event={"ID":"bc811bd0-2ab2-46b1-8d32-95cab341dcd7","Type":"ContainerStarted","Data":"c0137d73a8947a5c997a378274c2b27da4c046b460903b57b59363f7b4ae733b"} Oct 30 22:05:12.680024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:12.680003 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2229k" Oct 30 22:05:13.674663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:13.674625 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" event={"ID":"bc811bd0-2ab2-46b1-8d32-95cab341dcd7","Type":"ContainerStarted","Data":"050f19bed289aa2e284c3fad09c495011e93c6dd3c8dbb3ef88596345c7afd2a"} Oct 30 22:05:13.799546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:13.799488 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" podStartSLOduration=2.203073229 podStartE2EDuration="2.799476503s" totalImagesPullingTime="596.403274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:13.798126631 +0000 UTC m=+641.766946197" watchObservedRunningTime="2026-10-30 22:05:13.799476503 +0000 UTC m=+641.768296066" Oct 30 22:05:14.098603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:14.098547 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:15.377451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.377418 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:05:15.379724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.379706 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.434557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.434532 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 22:05:15.456254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456228 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456266 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456295 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456395 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456422 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456514 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxsld\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456586 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456635 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.456683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.456662 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.484345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.484318 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9qrfj\"" Oct 30 22:05:15.493300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.493271 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:05:15.557113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557078 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxsld\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557125 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557155 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557212 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557418 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557464 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557545 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557621 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557622 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557654 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557657 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.557901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.557879 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.558067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.558051 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.559448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.559426 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.559665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.559646 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.625205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.625180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.625840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.625825 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxsld\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\") pod \"maas-default-gateway-openshift-default-76597d6855-stk8v\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:15.690786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:15.690721 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:16.145634 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:16.145596 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81598a95_0804_4754_b8b3_c3328274f42f.slice/crio-4b6d77e711f89e4957fb698118bdc0ab2798a55241ba0ac52ead3cb3d7c2ddec WatchSource:0}: Error finding container 4b6d77e711f89e4957fb698118bdc0ab2798a55241ba0ac52ead3cb3d7c2ddec: Status 404 returned error can't find the container with id 4b6d77e711f89e4957fb698118bdc0ab2798a55241ba0ac52ead3cb3d7c2ddec Oct 30 22:05:16.145634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:16.145617 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:05:16.688662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:16.688626 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" event={"ID":"81598a95-0804-4754-b8b3-c3328274f42f","Type":"ContainerStarted","Data":"4b6d77e711f89e4957fb698118bdc0ab2798a55241ba0ac52ead3cb3d7c2ddec"} Oct 30 22:05:20.365490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:20.365458 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:20.365767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:20.365528 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:20.365767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:20.365558 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:20.708965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:20.708883 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" event={"ID":"81598a95-0804-4754-b8b3-c3328274f42f","Type":"ContainerStarted","Data":"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7"} Oct 30 22:05:20.935460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:20.935399 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podStartSLOduration=1.718285944 podStartE2EDuration="5.935384518s" totalImagesPullingTime="4.217098574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:20.884504346 +0000 UTC m=+648.853323924" watchObservedRunningTime="2026-10-30 22:05:20.935384518 +0000 UTC m=+648.904204081" Oct 30 22:05:21.701938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:21.701906 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:21.709720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:21.709684 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:21.709895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:21.709745 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:22.109600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:22.106939 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:22.120837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:22.120807 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:22.696597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:22.695874 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:22.696597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:22.695934 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:22.728209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:22.728182 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-hm5xf" Oct 30 22:05:23.701599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:23.693871 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:23.701599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:23.693944 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:24.697336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:24.697300 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:24.697951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:24.697916 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:25.698599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:25.696888 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:25.698599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:25.696954 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:26.639018 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:05:26.638979 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:05:26.691901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:26.691871 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.692036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:26.691933 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 22:05:27.467087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.467048 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wtllq"] Oct 30 22:05:27.469594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.469559 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.523967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.523938 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 22:05:27.566379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566346 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.566656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566395 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4241e8aa-cf0d-417e-8039-342452b62f8b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.566656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566461 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.566656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566506 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.566656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566523 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjql7\" (UniqueName: \"kubernetes.io/projected/4241e8aa-cf0d-417e-8039-342452b62f8b-kube-api-access-tjql7\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.566656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.566542 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.575008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.574980 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wtllq"] Oct 30 22:05:27.667682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667646 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.667827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667709 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.667827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667743 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjql7\" (UniqueName: \"kubernetes.io/projected/4241e8aa-cf0d-417e-8039-342452b62f8b-kube-api-access-tjql7\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.667827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667772 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.667967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667873 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.667967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.667926 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4241e8aa-cf0d-417e-8039-342452b62f8b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.668205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.668180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.668297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.668206 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.668297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.668260 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.669999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.669971 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4241e8aa-cf0d-417e-8039-342452b62f8b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.670710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.670689 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4241e8aa-cf0d-417e-8039-342452b62f8b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.734426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.734366 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjql7\" (UniqueName: \"kubernetes.io/projected/4241e8aa-cf0d-417e-8039-342452b62f8b-kube-api-access-tjql7\") pod \"echo-simulator-kserve-64b6f8f96c-wtllq\" (UID: \"4241e8aa-cf0d-417e-8039-342452b62f8b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:27.781058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:27.781032 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:28.162159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:28.162123 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wtllq"] Oct 30 22:05:28.163092 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:28.163062 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4241e8aa_cf0d_417e_8039_342452b62f8b.slice/crio-371334ae2425ffd59826d8cf97f823931bad39d64f6baf901152057071337266 WatchSource:0}: Error finding container 371334ae2425ffd59826d8cf97f823931bad39d64f6baf901152057071337266: Status 404 returned error can't find the container with id 371334ae2425ffd59826d8cf97f823931bad39d64f6baf901152057071337266 Oct 30 22:05:28.731743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:28.731700 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-stk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:28.732235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:28.731786 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:28.745564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:28.745526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" event={"ID":"4241e8aa-cf0d-417e-8039-342452b62f8b","Type":"ContainerStarted","Data":"371334ae2425ffd59826d8cf97f823931bad39d64f6baf901152057071337266"} Oct 30 22:05:29.016797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:29.016661 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:29.017119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:29.017090 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:29.018092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:29.018071 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:05:30.554106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.554075 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:05:30.557666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.557623 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593735 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593760 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593789 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593892 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593910 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.593949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593928 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.594067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.593963 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.594101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.594078 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.594134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.594122 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ccb\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.610384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.610341 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:05:30.694473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694491 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694512 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694532 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694600 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694650 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694695 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ccb\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.694852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694719 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694897 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694914 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.694961 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.695232 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.695274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.695676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.695654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.696928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.696908 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.696995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.696954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.757282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.757253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" event={"ID":"4241e8aa-cf0d-417e-8039-342452b62f8b","Type":"ContainerStarted","Data":"48c600337d9b80872e75468a30458ffe089aabf82a658a5cdf05cf5756f15ade"} Oct 30 22:05:30.757940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.757915 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.758382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.758361 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ccb\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\") pod \"maas-default-gateway-openshift-default-76597d6855-pw4rj\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.781569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.781552 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:30.870155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.870089 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:30.880003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:30.879965 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" podStartSLOduration=1.736269974 podStartE2EDuration="3.879953954s" totalImagesPullingTime="2.14368398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:30.878843325 +0000 UTC m=+658.847662890" watchObservedRunningTime="2026-10-30 22:05:30.879953954 +0000 UTC m=+658.848773518" Oct 30 22:05:31.296176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.296150 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:05:31.297498 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:31.297470 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff748f4_e860_4c3a_98f5_0e199853b238.slice/crio-9f52619b37863af00e24631d90036733e970364fae309c96a98b00bffaf56fd1 WatchSource:0}: Error finding container 9f52619b37863af00e24631d90036733e970364fae309c96a98b00bffaf56fd1: Status 404 returned error can't find the container with id 9f52619b37863af00e24631d90036733e970364fae309c96a98b00bffaf56fd1 Oct 30 22:05:31.300496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.300462 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:31.300615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.300542 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:31.300615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.300604 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:31.764626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.764511 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" event={"ID":"0ff748f4-e860-4c3a-98f5-0e199853b238","Type":"ContainerStarted","Data":"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57"} Oct 30 22:05:31.764626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.764607 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" event={"ID":"0ff748f4-e860-4c3a-98f5-0e199853b238","Type":"ContainerStarted","Data":"9f52619b37863af00e24631d90036733e970364fae309c96a98b00bffaf56fd1"} Oct 30 22:05:31.870987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:31.870956 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:32.872254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:32.872217 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:32.872727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:32.872288 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:33.870693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:33.870655 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.870859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:33.870710 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:34.871347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:34.871311 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:34.871868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:34.871399 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:35.870629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:35.870570 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:35.870806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:35.870652 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:36.870733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:36.870703 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:36.871185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:36.871160 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:37.782013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.781974 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:37.794626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.794594 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:37.805686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.805664 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-wtllq" Oct 30 22:05:37.870777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.870749 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-pw4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 22:05:37.871257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.870804 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 22:05:37.916919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:37.916865 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podStartSLOduration=7.916849623 podStartE2EDuration="7.916849623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:31.893498369 +0000 UTC m=+659.862317934" watchObservedRunningTime="2026-10-30 22:05:37.916849623 +0000 UTC m=+665.885669188" Oct 30 22:05:39.489406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:39.489378 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:39.489866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:39.489779 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:39.490422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:39.490400 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:05:51.209317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.209246 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:05:51.215102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.215083 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.272406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.272371 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:05:51.276420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.276395 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.276526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.276425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jj7\" (UniqueName: \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.320559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.320537 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 22:05:51.320661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.320595 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 22:05:51.320661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.320617 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nwtx4\"" Oct 30 22:05:51.377594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.377543 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.377700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.377611 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jj7\" (UniqueName: \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.379946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.379926 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.440305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.440285 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jj7\" (UniqueName: \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\") pod \"ai-gateway-controller-5d9cd99f45-lj5lq\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.527745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.527689 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:51.943765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.943738 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:05:51.945774 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:05:51.945751 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fc4828_c686_4489_a7b0_7dbf84c6570f.slice/crio-c320a4f42e46d16fcea13350f064ac380157985ef4123523cf04bae9b7f58aaa WatchSource:0}: Error finding container c320a4f42e46d16fcea13350f064ac380157985ef4123523cf04bae9b7f58aaa: Status 404 returned error can't find the container with id c320a4f42e46d16fcea13350f064ac380157985ef4123523cf04bae9b7f58aaa Oct 30 22:05:51.947052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:51.947029 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:05:52.855006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:52.854975 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" event={"ID":"52fc4828-c686-4489-a7b0-7dbf84c6570f","Type":"ContainerStarted","Data":"c320a4f42e46d16fcea13350f064ac380157985ef4123523cf04bae9b7f58aaa"} Oct 30 22:05:56.637842 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:05:56.637802 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:05:58.882274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:58.882237 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" event={"ID":"52fc4828-c686-4489-a7b0-7dbf84c6570f","Type":"ContainerStarted","Data":"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9"} Oct 30 22:05:58.882762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:58.882309 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:05:59.006026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:05:59.005979 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" podStartSLOduration=1.91840841 podStartE2EDuration="8.005965889s" totalImagesPullingTime="6.087557479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:59.004962669 +0000 UTC m=+686.973782235" watchObservedRunningTime="2026-10-30 22:05:59.005965889 +0000 UTC m=+686.974785447" Oct 30 22:06:01.267524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.267489 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p"] Oct 30 22:06:01.274125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.274106 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.331382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.331357 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p"] Oct 30 22:06:01.361167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.361142 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlrr\" (UniqueName: \"kubernetes.io/projected/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-kube-api-access-2wlrr\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.361265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.361182 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.379496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.379468 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 22:06:01.379666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.379515 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gs96h\"" Oct 30 22:06:01.432721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.432687 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z5c67"] Oct 30 22:06:01.436830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.436807 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.462543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.462511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlrr\" (UniqueName: \"kubernetes.io/projected/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-kube-api-access-2wlrr\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.462717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.462564 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.463689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.463667 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.493374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.493342 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z5c67"] Oct 30 22:06:01.526025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.525957 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlrr\" (UniqueName: \"kubernetes.io/projected/f5bb2c18-fe0f-4413-a332-c6d5b9db907b-kube-api-access-2wlrr\") pod \"payload-pre-processing-7b98bb678c-fgm2p\" (UID: \"f5bb2c18-fe0f-4413-a332-c6d5b9db907b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.563776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.563736 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f02934-7bd1-445e-96c0-0f0a740aeaef-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.563937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.563784 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnvh\" (UniqueName: \"kubernetes.io/projected/81f02934-7bd1-445e-96c0-0f0a740aeaef-kube-api-access-wtnvh\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.586708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.586678 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:01.665653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.665274 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f02934-7bd1-445e-96c0-0f0a740aeaef-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.665653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.665331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnvh\" (UniqueName: \"kubernetes.io/projected/81f02934-7bd1-445e-96c0-0f0a740aeaef-kube-api-access-wtnvh\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.666071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.666050 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f02934-7bd1-445e-96c0-0f0a740aeaef-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.729229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.729197 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnvh\" (UniqueName: \"kubernetes.io/projected/81f02934-7bd1-445e-96c0-0f0a740aeaef-kube-api-access-wtnvh\") pod \"payload-processing-55fbbbf9dc-z5c67\" (UID: \"81f02934-7bd1-445e-96c0-0f0a740aeaef\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.750700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.750672 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:01.969506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:01.969477 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p"] Oct 30 22:06:01.971098 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:06:01.971068 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bb2c18_fe0f_4413_a332_c6d5b9db907b.slice/crio-d636b160061559b43a44f831c3f2ee178d7e973f1c71284ff35b7b253094ca29 WatchSource:0}: Error finding container d636b160061559b43a44f831c3f2ee178d7e973f1c71284ff35b7b253094ca29: Status 404 returned error can't find the container with id d636b160061559b43a44f831c3f2ee178d7e973f1c71284ff35b7b253094ca29 Oct 30 22:06:02.137316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:02.137287 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z5c67"] Oct 30 22:06:02.139175 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:06:02.139145 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f02934_7bd1_445e_96c0_0f0a740aeaef.slice/crio-bac8404c6e1f2307a1af5903291ba829c18acdca84105dd32314493385a5b0b3 WatchSource:0}: Error finding container bac8404c6e1f2307a1af5903291ba829c18acdca84105dd32314493385a5b0b3: Status 404 returned error can't find the container with id bac8404c6e1f2307a1af5903291ba829c18acdca84105dd32314493385a5b0b3 Oct 30 22:06:02.902963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:02.902910 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" event={"ID":"81f02934-7bd1-445e-96c0-0f0a740aeaef","Type":"ContainerStarted","Data":"bac8404c6e1f2307a1af5903291ba829c18acdca84105dd32314493385a5b0b3"} Oct 30 22:06:02.904185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:02.904161 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" event={"ID":"f5bb2c18-fe0f-4413-a332-c6d5b9db907b","Type":"ContainerStarted","Data":"d636b160061559b43a44f831c3f2ee178d7e973f1c71284ff35b7b253094ca29"} Oct 30 22:06:05.922054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:05.922014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" event={"ID":"81f02934-7bd1-445e-96c0-0f0a740aeaef","Type":"ContainerStarted","Data":"0e35e7bdac0b860841cc9a36f84e828b4716cdf0f2c1f5b6ea1a978c47e6ba94"} Oct 30 22:06:05.922564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:05.922089 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:05.923764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:05.923735 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" event={"ID":"f5bb2c18-fe0f-4413-a332-c6d5b9db907b","Type":"ContainerStarted","Data":"1ad4e4fe1ec2c26ed370c4cbce162bb33db1ed0824c6141951d8f154f0692373"} Oct 30 22:06:05.923866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:05.923792 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:06.042863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:06.042813 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" podStartSLOduration=2.189978405 podStartE2EDuration="5.042799743s" totalImagesPullingTime="2.852821338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:06.041087486 +0000 UTC m=+694.009907052" watchObservedRunningTime="2026-10-30 22:06:06.042799743 +0000 UTC m=+694.011619330" Oct 30 22:06:06.162138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:06.162086 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" podStartSLOduration=2.169553922 podStartE2EDuration="5.162073148s" totalImagesPullingTime="2.992519226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:06.161064655 +0000 UTC m=+694.129884219" watchObservedRunningTime="2026-10-30 22:06:06.162073148 +0000 UTC m=+694.130892711" Oct 30 22:06:09.888977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:09.888947 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:06:10.268589 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.268510 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-858898f5f7-hp4sf"] Oct 30 22:06:10.272105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.272089 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.331993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.331967 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-858898f5f7-hp4sf"] Oct 30 22:06:10.446444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.446415 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-metrics-certs\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.446614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.446472 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5fpk\" (UniqueName: \"kubernetes.io/projected/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-kube-api-access-l5fpk\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.547297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.547269 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-metrics-certs\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.547408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.547342 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5fpk\" (UniqueName: \"kubernetes.io/projected/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-kube-api-access-l5fpk\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.549664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.549638 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-metrics-certs\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.610692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.610664 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5fpk\" (UniqueName: \"kubernetes.io/projected/48fe22ae-e5a2-476f-96ce-b4bbeb7fece3-kube-api-access-l5fpk\") pod \"ai-gateway-controller-858898f5f7-hp4sf\" (UID: \"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3\") " pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:10.885034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:10.884949 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:11.256538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:11.256508 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-858898f5f7-hp4sf"] Oct 30 22:06:11.257904 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:06:11.257879 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fe22ae_e5a2_476f_96ce_b4bbeb7fece3.slice/crio-6ac1f71d3492729c2515902a32d829b9171233cd106c1391ce7939c75ea9416a WatchSource:0}: Error finding container 6ac1f71d3492729c2515902a32d829b9171233cd106c1391ce7939c75ea9416a: Status 404 returned error can't find the container with id 6ac1f71d3492729c2515902a32d829b9171233cd106c1391ce7939c75ea9416a Oct 30 22:06:11.951837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:11.951798 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" event={"ID":"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3","Type":"ContainerStarted","Data":"6ac1f71d3492729c2515902a32d829b9171233cd106c1391ce7939c75ea9416a"} Oct 30 22:06:12.958479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:12.958441 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" event={"ID":"48fe22ae-e5a2-476f-96ce-b4bbeb7fece3","Type":"ContainerStarted","Data":"552e8f81da590c559d6f01b70dc6ac7e5123ced021385fd32711eeb5657599fb"} Oct 30 22:06:12.958985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:12.958508 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:13.084049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:13.083984 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" podStartSLOduration=2.131847018 podStartE2EDuration="3.083967526s" totalImagesPullingTime="952.120508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:13.081884701 +0000 UTC m=+701.050704266" watchObservedRunningTime="2026-10-30 22:06:13.083967526 +0000 UTC m=+701.052787091" Oct 30 22:06:16.931214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:16.931185 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fgm2p" Oct 30 22:06:16.931680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:16.931249 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z5c67" Oct 30 22:06:23.964600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:23.964548 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-858898f5f7-hp4sf" Oct 30 22:06:24.138411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.138379 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:06:24.138644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.138606 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" podUID="52fc4828-c686-4489-a7b0-7dbf84c6570f" containerName="manager" containerID="cri-o://19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9" gracePeriod=30 Oct 30 22:06:24.384483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.384457 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:06:24.468221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.468194 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\") pod \"52fc4828-c686-4489-a7b0-7dbf84c6570f\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " Oct 30 22:06:24.468379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.468282 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\" (UniqueName: \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\") pod \"52fc4828-c686-4489-a7b0-7dbf84c6570f\" (UID: \"52fc4828-c686-4489-a7b0-7dbf84c6570f\") " Oct 30 22:06:24.470449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.470415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7" pod "52fc4828-c686-4489-a7b0-7dbf84c6570f" (UID: "52fc4828-c686-4489-a7b0-7dbf84c6570f"). InnerVolumeSpecName "kube-api-access-88jj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:24.470552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.470446 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs" pod "52fc4828-c686-4489-a7b0-7dbf84c6570f" (UID: "52fc4828-c686-4489-a7b0-7dbf84c6570f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:24.569199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.569170 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88jj7\" (UniqueName: \"kubernetes.io/projected/52fc4828-c686-4489-a7b0-7dbf84c6570f-kube-api-access-88jj7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:24.569199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:24.569197 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fc4828-c686-4489-a7b0-7dbf84c6570f-metrics-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:25.010231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.010150 4292 generic.go:356] "Generic (PLEG): container finished" podID="52fc4828-c686-4489-a7b0-7dbf84c6570f" containerID="19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9" exitCode=0 Oct 30 22:06:25.010231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.010193 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" Oct 30 22:06:25.010649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.010240 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" event={"ID":"52fc4828-c686-4489-a7b0-7dbf84c6570f","Type":"ContainerDied","Data":"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9"} Oct 30 22:06:25.010649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.010318 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq" event={"ID":"52fc4828-c686-4489-a7b0-7dbf84c6570f","Type":"ContainerDied","Data":"c320a4f42e46d16fcea13350f064ac380157985ef4123523cf04bae9b7f58aaa"} Oct 30 22:06:25.010649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.010342 4292 scope.go:122] "RemoveContainer" containerID="19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9" Oct 30 22:06:25.018943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.018925 4292 scope.go:122] "RemoveContainer" containerID="19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9" Oct 30 22:06:25.019210 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:06:25.019192 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9\": container with ID starting with 19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9 not found: ID does not exist" containerID="19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9" Oct 30 22:06:25.019274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.019217 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9"} err="failed to get container status \"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9\": rpc error: code = NotFound desc = could not find container \"19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9\": container with ID starting with 19c43ad3d0165a3b84e08eabf6df6d2f1721e82d2984ff8f84d0118ee1df1bf9 not found: ID does not exist" Oct 30 22:06:25.178723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.178696 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:06:25.229144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:25.229118 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lj5lq"] Oct 30 22:06:26.145465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:26.145423 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fc4828-c686-4489-a7b0-7dbf84c6570f" path="/var/lib/kubelet/pods/52fc4828-c686-4489-a7b0-7dbf84c6570f/volumes" Oct 30 22:06:26.638258 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:06:26.638218 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:06:56.638474 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:06:56.638429 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:06:58.403127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.403089 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:06:58.403499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.403489 4292 cpu_manager.go:414] "containerMap: removing container" podUID="52fc4828-c686-4489-a7b0-7dbf84c6570f" containerName="manager" Oct 30 22:06:58.403540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.403499 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fc4828-c686-4489-a7b0-7dbf84c6570f" containerName="manager" Oct 30 22:06:58.403611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.403600 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fc4828-c686-4489-a7b0-7dbf84c6570f" containerName="manager" Oct 30 22:06:58.406556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.406539 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.472330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.472297 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2a4123-openshift-default-dockercfg-ldqs2\"" Oct 30 22:06:58.513127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.513095 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:06:58.567264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567237 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnz54\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567274 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567304 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567386 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567405 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567469 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567605 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567652 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.567827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.567711 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669115 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669147 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669190 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669249 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnz54\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669327 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669376 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669521 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.669899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.669872 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.670031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.670006 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.670105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.670088 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.670298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.670275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.671854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.671826 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.672168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.672150 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.732242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.732215 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnz54\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:58.732665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:58.732642 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:59.019043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.018947 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:06:59.068362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.068323 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:06:59.073285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.073261 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.127740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.127712 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-075cbb-openshift-default-dockercfg-88pqw\"" Oct 30 22:06:59.137123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.137087 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:06:59.172981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.172949 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173033 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173087 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173116 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173134 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173162 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173244 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173260 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.173343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.173316 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczqc\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.274790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274716 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.274790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.274790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274780 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274826 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274864 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.274978 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275035 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczqc\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275135 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275150 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275184 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275239 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275240 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275288 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.275873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.275849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.277315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.277287 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.277558 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.277543 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.337857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.337835 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.338003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.337982 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczqc\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.396164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.396135 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:06:59.450474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.450441 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:06:59.455998 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:06:59.455969 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86002a59_1b82_4516_8b86_c3251d6936cb.slice/crio-0cc239777507eb4315c5ce632cdc0dda12ab903ac9a803cde075fcaf560f2205 WatchSource:0}: Error finding container 0cc239777507eb4315c5ce632cdc0dda12ab903ac9a803cde075fcaf560f2205: Status 404 returned error can't find the container with id 0cc239777507eb4315c5ce632cdc0dda12ab903ac9a803cde075fcaf560f2205 Oct 30 22:06:59.462301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.461160 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.462301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.461226 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.462301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.461259 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.748530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.748500 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:06:59.752198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.752177 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.795965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.795680 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:06:59.796200 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:06:59.796173 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c29bf03_9fd9_4a32_8cb7_ae6dd3021b8a.slice/crio-824a9e1a092113f2b05118479ec2b170fcda1c6db0f343b17b35a7af8d346e9b WatchSource:0}: Error finding container 824a9e1a092113f2b05118479ec2b170fcda1c6db0f343b17b35a7af8d346e9b: Status 404 returned error can't find the container with id 824a9e1a092113f2b05118479ec2b170fcda1c6db0f343b17b35a7af8d346e9b Oct 30 22:06:59.799711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.799670 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.799806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.799750 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.799806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.799791 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:59.807970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.807949 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-97aa64-openshift-default-dockercfg-dhxrl\"" Oct 30 22:06:59.815315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.815291 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:06:59.884475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884447 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2pr\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884494 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884566 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884665 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884709 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884747 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884840 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884876 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.884988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.884939 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.985983 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2pr\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986022 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986042 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986089 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986117 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986151 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986228 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986336 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986699 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986812 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.986865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986802 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.987125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.986894 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.987173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.987120 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.988553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.988532 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:06:59.988682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:06:59.988561 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:00.049218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.049177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:00.049367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.049230 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2pr\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-jhx79\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:00.064824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.064796 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:00.093522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.093485 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:07:00.098196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.098176 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.167273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.167231 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerStarted","Data":"157724133e1d3f23b74adef15d9fccfcb5faee18e1d1f8b924a51d00144c1f5e"} Oct 30 22:07:00.167524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.167503 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerStarted","Data":"824a9e1a092113f2b05118479ec2b170fcda1c6db0f343b17b35a7af8d346e9b"} Oct 30 22:07:00.170468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.170437 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerStarted","Data":"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158"} Oct 30 22:07:00.170596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.170489 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerStarted","Data":"0cc239777507eb4315c5ce632cdc0dda12ab903ac9a803cde075fcaf560f2205"} Oct 30 22:07:00.172484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.172461 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:07:00.230875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.230846 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-963c3a-openshift-default-dockercfg-fd9jr\"" Oct 30 22:07:00.289791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.289757 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.289791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.289791 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290204 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290410 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290548 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.290691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.290630 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lp8g\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.343779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.343701 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podStartSLOduration=2.343690317 podStartE2EDuration="2.343690317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:00.341620698 +0000 UTC m=+748.310440264" watchObservedRunningTime="2026-10-30 22:07:00.343690317 +0000 UTC m=+748.312509881" Oct 30 22:07:00.391546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391516 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391548 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391602 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391625 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391801 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391856 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391875 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lp8g\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391920 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.391940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391939 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.392206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391947 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.392206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.391967 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.392206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.392109 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.392392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.392246 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.392520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.392500 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.394010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.393984 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.394163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.394148 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.396342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.396323 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:00.401687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.401670 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:00.455017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.454961 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lp8g\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.455391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.455088 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:00.456867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.456848 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:07:00.459347 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:00.459322 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdd9e96_cc13_4fcd_9e95_0a3cfa08a321.slice/crio-44098fa21e14e4f97e4bc4ae6a9a752b8027ef0e2bac463a036c3770069d88bf WatchSource:0}: Error finding container 44098fa21e14e4f97e4bc4ae6a9a752b8027ef0e2bac463a036c3770069d88bf: Status 404 returned error can't find the container with id 44098fa21e14e4f97e4bc4ae6a9a752b8027ef0e2bac463a036c3770069d88bf Oct 30 22:07:00.461527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.461502 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:00.461596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.461561 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:00.461636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.461605 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:00.467665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.467625 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podStartSLOduration=1.467613549 podStartE2EDuration="1.467613549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:00.465684041 +0000 UTC m=+748.434503607" watchObservedRunningTime="2026-10-30 22:07:00.467613549 +0000 UTC m=+748.436433112" Oct 30 22:07:00.713786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:00.713712 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:01.019171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.019095 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:07:01.025315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.025290 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:07:01.098161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.098137 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:07:01.099468 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:01.099438 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880c84c7_7c67_4501_9f05_f3256c11d4e1.slice/crio-b62d259ef7e25f6eb6c76c7509d48bc452d34b91503adb50bd73077dd64b1bd2 WatchSource:0}: Error finding container b62d259ef7e25f6eb6c76c7509d48bc452d34b91503adb50bd73077dd64b1bd2: Status 404 returned error can't find the container with id b62d259ef7e25f6eb6c76c7509d48bc452d34b91503adb50bd73077dd64b1bd2 Oct 30 22:07:01.101744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.101715 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:01.101821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.101780 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:01.101821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.101810 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:01.178619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.177409 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerStarted","Data":"f3e58cd5021bca2f876a05a1835a4f348862e4620be7bb4857ff3178fb147e33"} Oct 30 22:07:01.178619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.177470 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerStarted","Data":"b62d259ef7e25f6eb6c76c7509d48bc452d34b91503adb50bd73077dd64b1bd2"} Oct 30 22:07:01.179457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.179425 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerStarted","Data":"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013"} Oct 30 22:07:01.179592 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.179484 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerStarted","Data":"44098fa21e14e4f97e4bc4ae6a9a752b8027ef0e2bac463a036c3770069d88bf"} Oct 30 22:07:01.179967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.179946 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:07:01.180086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.179992 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:01.180911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.180889 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:01.182347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.182328 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:07:01.303270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.303226 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podStartSLOduration=1.303211578 podStartE2EDuration="1.303211578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:01.301362554 +0000 UTC m=+749.270182120" watchObservedRunningTime="2026-10-30 22:07:01.303211578 +0000 UTC m=+749.272031139" Oct 30 22:07:01.559647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.559477 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podStartSLOduration=2.559461583 podStartE2EDuration="2.559461583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:01.432905632 +0000 UTC m=+749.401725235" watchObservedRunningTime="2026-10-30 22:07:01.559461583 +0000 UTC m=+749.528281147" Oct 30 22:07:01.714474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:01.714435 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:02.065648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:02.065619 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:02.070594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:02.070546 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:02.185029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:02.184995 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:02.186036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:02.186014 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:07:02.719355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:02.719323 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:03.189585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:03.189545 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:03.190143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:03.190122 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:07:06.579533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.579500 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:07:06.584247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.584230 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.637646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.637615 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-77cb1f-dockercfg-gdvfz\"" Oct 30 22:07:06.654934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.654906 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.655122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.655096 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hwd\" (UniqueName: \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.688136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.688105 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-77cb1f\"" Oct 30 22:07:06.690016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.689994 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:07:06.756201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.756172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hwd\" (UniqueName: \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.756346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.756217 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.756975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.756949 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.819072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.819052 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hwd\" (UniqueName: \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\") pod \"payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:06.897444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:06.897375 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:07.276612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:07.276565 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:07:07.277976 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:07.277935 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436426e1_cc3f_4212_9b7c_21bba88d61e1.slice/crio-61383ac686c2d300c5dc506468811f8b294a363880184b39770120f0c7f31ab3 WatchSource:0}: Error finding container 61383ac686c2d300c5dc506468811f8b294a363880184b39770120f0c7f31ab3: Status 404 returned error can't find the container with id 61383ac686c2d300c5dc506468811f8b294a363880184b39770120f0c7f31ab3 Oct 30 22:07:08.213820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:08.213780 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" event={"ID":"436426e1-cc3f-4212-9b7c-21bba88d61e1","Type":"ContainerStarted","Data":"7488f2acc2261f0533b9059774232f857448a9c827cd41b32e435f3ecb3450d2"} Oct 30 22:07:08.214267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:08.213884 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:08.214267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:08.213901 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" event={"ID":"436426e1-cc3f-4212-9b7c-21bba88d61e1","Type":"ContainerStarted","Data":"61383ac686c2d300c5dc506468811f8b294a363880184b39770120f0c7f31ab3"} Oct 30 22:07:08.355530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:08.355484 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" podStartSLOduration=2.355469357 podStartE2EDuration="2.355469357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:08.354055037 +0000 UTC m=+756.322874602" watchObservedRunningTime="2026-10-30 22:07:08.355469357 +0000 UTC m=+756.324288924" Oct 30 22:07:18.499953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.499917 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:07:18.511885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.511863 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.563226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.563197 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:07:18.582388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.582361 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.582514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.582426 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsth\" (UniqueName: \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.610948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.610925 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2a4123\"" Oct 30 22:07:18.611037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.610967 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2a4123-dockercfg-bbg6l\"" Oct 30 22:07:18.683979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.683945 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.684120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.684030 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsth\" (UniqueName: \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.684706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.684685 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.747600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.747538 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsth\" (UniqueName: \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\") pod \"payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:18.823746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:18.823719 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:19.197138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.197113 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:07:19.198738 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:19.198708 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d8fe97_233f_4841_9d5b_51255fc08384.slice/crio-3ad61de9160131ae3f397d7f73ce77fb85617ea382797d5c637b1ede65282479 WatchSource:0}: Error finding container 3ad61de9160131ae3f397d7f73ce77fb85617ea382797d5c637b1ede65282479: Status 404 returned error can't find the container with id 3ad61de9160131ae3f397d7f73ce77fb85617ea382797d5c637b1ede65282479 Oct 30 22:07:19.221342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.221322 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:07:19.266978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.266943 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" event={"ID":"30d8fe97-233f-4841-9d5b-51255fc08384","Type":"ContainerStarted","Data":"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924"} Oct 30 22:07:19.267075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.267013 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" event={"ID":"30d8fe97-233f-4841-9d5b-51255fc08384","Type":"ContainerStarted","Data":"3ad61de9160131ae3f397d7f73ce77fb85617ea382797d5c637b1ede65282479"} Oct 30 22:07:19.267075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.267040 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:19.465684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:19.465540 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" podStartSLOduration=1.465527897 podStartE2EDuration="1.465527897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.464826008 +0000 UTC m=+767.433645572" watchObservedRunningTime="2026-10-30 22:07:19.465527897 +0000 UTC m=+767.434347514" Oct 30 22:07:21.076607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.076564 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:07:21.080443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.080420 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.138930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.138911 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-963c3a\"" Oct 30 22:07:21.188728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.188697 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-963c3a-dockercfg-g58qq\"" Oct 30 22:07:21.191261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.191238 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:07:21.211457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.211205 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.211457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.211337 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4l4\" (UniqueName: \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.241024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.240999 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:07:21.244467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.244451 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.299107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.299082 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:07:21.312221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.312194 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.312330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.312230 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4l4\" (UniqueName: \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.312330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.312255 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.312455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.312334 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsf8\" (UniqueName: \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.312885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.312863 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.375136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.375077 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4l4\" (UniqueName: \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\") pod \"payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.392888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.392865 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:21.413226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.413204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.413324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.413301 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsf8\" (UniqueName: \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.413883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.413865 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.479057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.479026 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsf8\" (UniqueName: \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\") pod \"payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.556699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.556667 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:21.770296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.770272 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:07:21.771950 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:21.771922 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992b0a74_db4f_450e_a47c_1d420001bf95.slice/crio-3ce4b89584c36a16bcf23cf02c84b6fde290d6e2fb254f1dd1cbf6a15b70528e WatchSource:0}: Error finding container 3ce4b89584c36a16bcf23cf02c84b6fde290d6e2fb254f1dd1cbf6a15b70528e: Status 404 returned error can't find the container with id 3ce4b89584c36a16bcf23cf02c84b6fde290d6e2fb254f1dd1cbf6a15b70528e Oct 30 22:07:21.943520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:21.943490 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:07:21.944926 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:21.944905 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9370076_ea62_4fcb_96f0_f8efd710e280.slice/crio-9f04f854c3ae5b9670fc45144d61a3e34bce2bd939a608bb2477816d0c9c1890 WatchSource:0}: Error finding container 9f04f854c3ae5b9670fc45144d61a3e34bce2bd939a608bb2477816d0c9c1890: Status 404 returned error can't find the container with id 9f04f854c3ae5b9670fc45144d61a3e34bce2bd939a608bb2477816d0c9c1890 Oct 30 22:07:22.284980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.284941 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" event={"ID":"b9370076-ea62-4fcb-96f0-f8efd710e280","Type":"ContainerStarted","Data":"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572"} Oct 30 22:07:22.285467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.285001 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" event={"ID":"b9370076-ea62-4fcb-96f0-f8efd710e280","Type":"ContainerStarted","Data":"9f04f854c3ae5b9670fc45144d61a3e34bce2bd939a608bb2477816d0c9c1890"} Oct 30 22:07:22.285467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.285024 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:22.286622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.286592 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" event={"ID":"992b0a74-db4f-450e-a47c-1d420001bf95","Type":"ContainerStarted","Data":"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0"} Oct 30 22:07:22.286757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.286637 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" event={"ID":"992b0a74-db4f-450e-a47c-1d420001bf95","Type":"ContainerStarted","Data":"3ce4b89584c36a16bcf23cf02c84b6fde290d6e2fb254f1dd1cbf6a15b70528e"} Oct 30 22:07:22.286757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.286659 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:22.405366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.405318 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" podStartSLOduration=1.405303818 podStartE2EDuration="1.405303818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:22.403344609 +0000 UTC m=+770.372164173" watchObservedRunningTime="2026-10-30 22:07:22.405303818 +0000 UTC m=+770.374123376" Oct 30 22:07:22.523030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:22.522981 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" podStartSLOduration=1.5229679950000001 podStartE2EDuration="1.522967995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:22.521631926 +0000 UTC m=+770.490451489" watchObservedRunningTime="2026-10-30 22:07:22.522967995 +0000 UTC m=+770.491787561" Oct 30 22:07:23.717496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.717458 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:07:23.721189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.721172 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:23.780297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.780261 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:07:23.828383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.828353 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-97aa64-dockercfg-pbtgm\"" Oct 30 22:07:23.828383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.828361 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-97aa64\"" Oct 30 22:07:23.838389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.838361 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtxl\" (UniqueName: \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:23.838554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.838536 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:23.939544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.939511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtxl\" (UniqueName: \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:23.939689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.939656 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:23.940750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:23.940730 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:24.002890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:24.002834 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtxl\" (UniqueName: \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\") pod \"payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:24.038354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:24.038328 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:24.409697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:24.409672 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:07:24.411070 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:24.411042 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c8267c_cca3_4e4d_8b41_4291130d9e97.slice/crio-1ee456924cde29710f565a27818bc2e2dedec4401ced1366766f77a1b92a81e7 WatchSource:0}: Error finding container 1ee456924cde29710f565a27818bc2e2dedec4401ced1366766f77a1b92a81e7: Status 404 returned error can't find the container with id 1ee456924cde29710f565a27818bc2e2dedec4401ced1366766f77a1b92a81e7 Oct 30 22:07:25.302915 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:25.302883 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" event={"ID":"40c8267c-cca3-4e4d-8b41-4291130d9e97","Type":"ContainerStarted","Data":"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e"} Oct 30 22:07:25.303280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:25.302932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" event={"ID":"40c8267c-cca3-4e4d-8b41-4291130d9e97","Type":"ContainerStarted","Data":"1ee456924cde29710f565a27818bc2e2dedec4401ced1366766f77a1b92a81e7"} Oct 30 22:07:25.303280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:25.302996 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:25.422124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:25.422081 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" podStartSLOduration=2.42206923 podStartE2EDuration="2.42206923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:25.421750199 +0000 UTC m=+773.390569762" watchObservedRunningTime="2026-10-30 22:07:25.42206923 +0000 UTC m=+773.390888794" Oct 30 22:07:26.119747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.119710 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:07:26.123463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.123447 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.180611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.180557 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-075cbb\"" Oct 30 22:07:26.182482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.182458 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:07:26.238714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.230482 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-075cbb-dockercfg-znrdq\"" Oct 30 22:07:26.262198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.262162 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.262376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.262212 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p5w\" (UniqueName: \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.363679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.363638 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.364021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.363687 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p5w\" (UniqueName: \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.364301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.364281 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.427189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.427125 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p5w\" (UniqueName: \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\") pod \"payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.435759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.435735 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:26.638705 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:07:26.638670 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:07:26.807854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:26.807827 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:07:26.809253 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:26.809225 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3486b1_926d_4d61_9eae_7d62a3179d85.slice/crio-52858bdbb916ddb2b174e00320bfc0e06575e12f8393556dd64d1e756d096fb4 WatchSource:0}: Error finding container 52858bdbb916ddb2b174e00320bfc0e06575e12f8393556dd64d1e756d096fb4: Status 404 returned error can't find the container with id 52858bdbb916ddb2b174e00320bfc0e06575e12f8393556dd64d1e756d096fb4 Oct 30 22:07:27.314265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:27.314229 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" event={"ID":"3f3486b1-926d-4d61-9eae-7d62a3179d85","Type":"ContainerStarted","Data":"969468c3c5d00ea3402725a15d9fa11bc6edc47d75c3ae33330bca99ad4df64a"} Oct 30 22:07:27.314433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:27.314282 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" event={"ID":"3f3486b1-926d-4d61-9eae-7d62a3179d85","Type":"ContainerStarted","Data":"52858bdbb916ddb2b174e00320bfc0e06575e12f8393556dd64d1e756d096fb4"} Oct 30 22:07:27.314433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:27.314313 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:27.435323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:27.435266 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" podStartSLOduration=1.435253468 podStartE2EDuration="1.435253468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:27.434258675 +0000 UTC m=+775.403078240" watchObservedRunningTime="2026-10-30 22:07:27.435253468 +0000 UTC m=+775.404073032" Oct 30 22:07:30.277446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:30.277416 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:07:33.295782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:33.295751 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:07:33.296230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:33.295908 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:07:36.311766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.311741 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:07:36.369443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.369412 4292 generic.go:356] "Generic (PLEG): container finished" podID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerID="157724133e1d3f23b74adef15d9fccfcb5faee18e1d1f8b924a51d00144c1f5e" exitCode=0 Oct 30 22:07:36.369635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.369492 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerDied","Data":"157724133e1d3f23b74adef15d9fccfcb5faee18e1d1f8b924a51d00144c1f5e"} Oct 30 22:07:36.370079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.370064 4292 scope.go:122] "RemoveContainer" containerID="157724133e1d3f23b74adef15d9fccfcb5faee18e1d1f8b924a51d00144c1f5e" Oct 30 22:07:36.371933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.371848 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:36.372045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.372021 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:36.372100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:36.372073 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:37.376155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:37.376114 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerStarted","Data":"63562757b4f9e233a4d6bcb02a261f5298ba45d2c4420cf70e7ff0bcfc4b22ce"} Oct 30 22:07:37.406596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:37.398995 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:37.406596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:37.399120 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:37.406596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:37.399176 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:38.322644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:38.322617 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:07:38.404609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:38.403895 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:38.404609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:38.403951 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:39.399968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:39.399931 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:39.400231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:39.400204 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:40.401386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:40.400943 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:40.401386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:40.401016 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:41.405598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:41.403945 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:41.405598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:41.404025 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:42.396484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:42.396441 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 22:07:42.396683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:42.396493 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 22:07:43.401030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:43.400997 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:43.401420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:43.401325 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:43.402038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:43.402018 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:07:44.881487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.881456 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:07:44.886766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.886745 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.941136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.941115 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-13b8d8-openshift-default-dockercfg-8442c\"" Oct 30 22:07:44.991107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991109 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991182 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswmw\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991233 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991251 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991308 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991418 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991486 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.991551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.991540 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:44.993131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:44.993111 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:07:45.092943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.092910 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.092981 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093038 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093097 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswmw\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093127 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093153 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093329 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093516 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093410 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093547 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093793 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093818 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.093888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.093818 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.095435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.095414 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.095567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.095549 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.156313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.156246 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.156452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.156436 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswmw\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.200151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.200125 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:45.627686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.627652 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:07:45.660447 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:45.660411 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc5c76c_3473_4619_a66a_096f1e769f67.slice/crio-7d30f3319c31d75178d7003c560e7151575b73764f020751b5322c925f24328e WatchSource:0}: Error finding container 7d30f3319c31d75178d7003c560e7151575b73764f020751b5322c925f24328e: Status 404 returned error can't find the container with id 7d30f3319c31d75178d7003c560e7151575b73764f020751b5322c925f24328e Oct 30 22:07:45.662640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.662608 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:45.662731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.662679 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:45.662731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:45.662711 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:46.443077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:46.443036 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" event={"ID":"7fc5c76c-3473-4619-a66a-096f1e769f67","Type":"ContainerStarted","Data":"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80"} Oct 30 22:07:46.443644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:46.443090 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" event={"ID":"7fc5c76c-3473-4619-a66a-096f1e769f67","Type":"ContainerStarted","Data":"7d30f3319c31d75178d7003c560e7151575b73764f020751b5322c925f24328e"} Oct 30 22:07:46.580832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:46.580768 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podStartSLOduration=2.580750851 podStartE2EDuration="2.580750851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:46.579711706 +0000 UTC m=+794.548531270" watchObservedRunningTime="2026-10-30 22:07:46.580750851 +0000 UTC m=+794.549570416" Oct 30 22:07:47.210179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:47.209826 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:47.211530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:47.211498 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 22:07:47.211683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:47.211557 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 22:07:48.205598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:48.204667 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 22:07:48.205598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:48.204753 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 22:07:49.203073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:49.203032 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 22:07:49.203375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:49.203344 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 22:07:50.206599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:50.201980 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 22:07:50.206599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:50.202040 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 22:07:51.208599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:51.208001 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 22:07:51.208599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:51.208065 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 22:07:52.212863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:52.212828 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:52.213284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:52.213181 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:52.214061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:52.214042 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:07:53.734924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.734892 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:07:53.738506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.738489 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820430 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820514 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820626 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820679 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820776 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.820946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.820930 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvfx\" (UniqueName: \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.849399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.849372 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"default-dockercfg-s8b25\"" Oct 30 22:07:53.849548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.849400 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"simulator-w6-c76d9e-kserve-self-signed-certs\"" Oct 30 22:07:53.849548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.849439 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"kube-root-ca.crt\"" Oct 30 22:07:53.849548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.849465 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"openshift-service-ca.crt\"" Oct 30 22:07:53.851836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.851816 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:07:53.921924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.921891 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvfx\" (UniqueName: \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.921935 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922118 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922152 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922173 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922504 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922530 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.922678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.922552 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.924287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.924266 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:53.924781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:53.924762 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:54.007503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:54.007400 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvfx\" (UniqueName: \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\") pod \"simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:54.051014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:54.050984 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:54.457004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:54.456977 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:07:54.458610 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:07:54.458556 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439bff93_844c_41b5_ba0e_705bbf8db3a5.slice/crio-901a093d6e44385cbeb3d2070197a6039a922a4b151117bb7967dca8a802e56a WatchSource:0}: Error finding container 901a093d6e44385cbeb3d2070197a6039a922a4b151117bb7967dca8a802e56a: Status 404 returned error can't find the container with id 901a093d6e44385cbeb3d2070197a6039a922a4b151117bb7967dca8a802e56a Oct 30 22:07:54.501921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:54.501886 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" event={"ID":"439bff93-844c-41b5-ba0e-705bbf8db3a5","Type":"ContainerStarted","Data":"901a093d6e44385cbeb3d2070197a6039a922a4b151117bb7967dca8a802e56a"} Oct 30 22:07:55.509559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:55.509518 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" event={"ID":"439bff93-844c-41b5-ba0e-705bbf8db3a5","Type":"ContainerStarted","Data":"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c"} Oct 30 22:07:55.632996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:55.632936 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podStartSLOduration=2.632919676 podStartE2EDuration="2.632919676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:55.630542087 +0000 UTC m=+803.599361652" watchObservedRunningTime="2026-10-30 22:07:55.632919676 +0000 UTC m=+803.601739239" Oct 30 22:07:56.051867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:56.051823 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:07:56.640826 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:07:56.640787 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:07:58.525355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.525325 4292 generic.go:356] "Generic (PLEG): container finished" podID="86002a59-1b82-4516-8b86-c3251d6936cb" containerID="badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158" exitCode=0 Oct 30 22:07:58.525964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.525400 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerDied","Data":"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158"} Oct 30 22:07:58.525964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.525871 4292 scope.go:122] "RemoveContainer" containerID="badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158" Oct 30 22:07:58.527193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.527169 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:58.527254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.527233 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:58.527288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:58.527263 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:59.539478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:07:59.539445 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerStarted","Data":"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837"} Oct 30 22:08:00.024598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:00.023634 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:08:00.031601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:00.027692 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.031601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:00.027754 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 22:08:01.021056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:01.021017 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.021684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:01.021651 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 22:08:01.180565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:01.180539 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:08:02.023063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:02.023023 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 22:08:02.023701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:02.023668 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 22:08:03.020476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:03.020442 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 22:08:03.020642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:03.020504 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 22:08:04.020090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.020058 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 22:08:04.020522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.020109 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 22:08:04.051660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.051619 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:08:04.066454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.066424 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:08:04.245718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.245682 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:04.245903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.245734 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:04.576666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:04.576634 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:08:05.024011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:05.023936 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:08:05.025211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:05.025190 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:08:07.583592 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.583551 4292 generic.go:356] "Generic (PLEG): container finished" podID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerID="f3e58cd5021bca2f876a05a1835a4f348862e4620be7bb4857ff3178fb147e33" exitCode=0 Oct 30 22:08:07.584033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.583624 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerDied","Data":"f3e58cd5021bca2f876a05a1835a4f348862e4620be7bb4857ff3178fb147e33"} Oct 30 22:08:07.584033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.583982 4292 scope.go:122] "RemoveContainer" containerID="f3e58cd5021bca2f876a05a1835a4f348862e4620be7bb4857ff3178fb147e33" Oct 30 22:08:07.585203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.585179 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:07.585255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.585245 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:07.585293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:07.585278 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:08.638018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:08.637976 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerStarted","Data":"486543175baf4084f7d1e33d3ca2b0037571608dca8d51e14d44b14dcf485acf"} Oct 30 22:08:08.723600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:08.715016 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:08:08.723600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:08.715148 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:08.723600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:08.715193 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:09.723869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:09.723831 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:09.724476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:09.724439 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:10.718008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:10.717969 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:10.718289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:10.718256 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:11.717001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:11.716961 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:11.718322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:11.718290 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:12.721651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:12.721613 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:12.722223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:12.722193 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:13.714845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:13.714808 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:08:13.715007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:13.714864 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:08:14.485019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.484973 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:08:14.488769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.488744 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.547115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.547081 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c76d9e-openshift-default-dockercfg-r2lmn\"" Oct 30 22:08:14.595909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.595876 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:08:14.647445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647402 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbsl\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647453 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647479 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647605 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647652 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647760 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647807 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.647896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647886 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.648042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.647948 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.720085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.720053 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:08:14.720829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.720801 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:08:14.721554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.721528 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:08:14.749180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749108 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749161 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749206 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749236 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749539 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749660 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749726 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbsl\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749771 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749799 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.749910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.749861 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.750206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.750179 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.750315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.750180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.750560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.750534 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.751658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.751636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.751988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.751968 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.815525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.815483 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbsl\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:14.815691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:14.815553 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:15.102568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.102530 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:15.533020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.532991 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:08:15.534267 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:08:15.534246 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d6e81a_de8c_45ed_be7b_93c2bacb3235.slice/crio-f3eb354ee25fb43a38f21f63267af7738a04ff82ff9ccde9d3b40f9361475831 WatchSource:0}: Error finding container f3eb354ee25fb43a38f21f63267af7738a04ff82ff9ccde9d3b40f9361475831: Status 404 returned error can't find the container with id f3eb354ee25fb43a38f21f63267af7738a04ff82ff9ccde9d3b40f9361475831 Oct 30 22:08:15.536596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.536544 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:15.536680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.536647 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:15.536718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.536679 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:15.682268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.682230 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" event={"ID":"c5d6e81a-de8c-45ed-be7b-93c2bacb3235","Type":"ContainerStarted","Data":"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313"} Oct 30 22:08:15.682447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.682294 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" event={"ID":"c5d6e81a-de8c-45ed-be7b-93c2bacb3235","Type":"ContainerStarted","Data":"f3eb354ee25fb43a38f21f63267af7738a04ff82ff9ccde9d3b40f9361475831"} Oct 30 22:08:15.810807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:15.810747 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podStartSLOduration=1.810729518 podStartE2EDuration="1.810729518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:15.807292932 +0000 UTC m=+823.776112500" watchObservedRunningTime="2026-10-30 22:08:15.810729518 +0000 UTC m=+823.779549085" Oct 30 22:08:16.105959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:16.105866 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:16.106289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:16.106216 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:16.106476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:16.106449 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:17.104341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.103893 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:17.104341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.103953 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:17.226814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.226766 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:59362->10.134.0.37:15021: read: connection reset by peer" start-of-body= Oct 30 22:08:17.227081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.226827 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:59362->10.134.0.37:15021: read: connection reset by peer" Oct 30 22:08:17.718782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.718738 4292 generic.go:356] "Generic (PLEG): container finished" podID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerID="ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013" exitCode=0 Oct 30 22:08:17.719752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.719715 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerDied","Data":"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013"} Oct 30 22:08:17.720239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.720217 4292 scope.go:122] "RemoveContainer" containerID="ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013" Oct 30 22:08:17.721677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.721634 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:17.721808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.721728 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:17.721808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:17.721762 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:18.109085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:18.109042 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:18.110597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:18.109706 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:18.752196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:18.752155 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerStarted","Data":"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2"} Oct 30 22:08:19.066032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:19.066005 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:08:19.066188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:19.066109 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:08:19.066188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:19.066139 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:08:19.104595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:19.103953 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:19.104595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:19.104010 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:20.065513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:20.065475 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:08:20.065942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:20.065539 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:08:20.110597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:20.108958 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:20.110597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:20.109019 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:21.066113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:21.066075 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:08:21.066497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:21.066132 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:08:21.103652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:21.103563 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:08:21.103883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:21.103855 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:08:22.066113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:22.066075 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:08:22.066292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:22.066154 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:08:22.111596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:22.111424 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:22.112319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:22.112289 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:22.112956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:22.112936 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:08:23.065962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:23.065926 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:08:23.066131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:23.065980 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:08:24.069368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:24.069339 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:08:24.069757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:24.069605 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:08:24.070439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:24.070424 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:08:26.639675 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:08:26.639637 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:08:47.862101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.862013 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:08:47.867507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.867484 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873546 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873671 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873715 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873811 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.873868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.873840 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcnt\" (UniqueName: \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.971798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.971771 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:08:47.973816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.973797 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"premium-w6-c76d9e-kserve-self-signed-certs\"" Oct 30 22:08:47.975255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975233 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975269 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcnt\" (UniqueName: \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975317 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975362 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975442 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975496 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975673 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975771 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.975874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.975852 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.977536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.977514 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:47.977847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:47.977829 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:48.040710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:48.040679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcnt\" (UniqueName: \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\") pod \"premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:48.213968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:48.213892 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:48.602964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:48.602930 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:08:48.604287 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:08:48.604261 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b33a316_fb2d_42e7_869e_fefc2700863f.slice/crio-60fec557aa16900316d16042a709f01ec52b01494720123e5f4002cf1780b3b6 WatchSource:0}: Error finding container 60fec557aa16900316d16042a709f01ec52b01494720123e5f4002cf1780b3b6: Status 404 returned error can't find the container with id 60fec557aa16900316d16042a709f01ec52b01494720123e5f4002cf1780b3b6 Oct 30 22:08:48.921081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:48.921010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" event={"ID":"7b33a316-fb2d-42e7-869e-fefc2700863f","Type":"ContainerStarted","Data":"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51"} Oct 30 22:08:48.921081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:48.921049 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" event={"ID":"7b33a316-fb2d-42e7-869e-fefc2700863f","Type":"ContainerStarted","Data":"60fec557aa16900316d16042a709f01ec52b01494720123e5f4002cf1780b3b6"} Oct 30 22:08:49.046026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:49.045984 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podStartSLOduration=2.045971834 podStartE2EDuration="2.045971834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:49.041009352 +0000 UTC m=+857.009828916" watchObservedRunningTime="2026-10-30 22:08:49.045971834 +0000 UTC m=+857.014791397" Oct 30 22:08:49.214321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:49.214249 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:56.646147 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:08:56.646105 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:08:58.214217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:58.214183 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:58.226554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:58.226529 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:08:58.989750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:08:58.989714 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:09:21.849761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.849728 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:09:21.853387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.853368 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:21.933588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.933539 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:09:21.961442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.961413 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-97aa64\"/\"kube-root-ca.crt\"" Oct 30 22:09:21.961442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.961434 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-97aa64\"/\"openshift-service-ca.crt\"" Oct 30 22:09:21.961634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.961489 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-97aa64\"/\"default-dockercfg-v57fc\"" Oct 30 22:09:21.961803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:21.961790 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-97aa64\"/\"simulator-w4-97aa64-kserve-self-signed-certs\"" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018401 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018464 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018521 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018551 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wjr\" (UniqueName: \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018610 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.019889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.018786 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119616 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119653 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119682 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119703 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wjr\" (UniqueName: \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119720 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.119949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.119760 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.120124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.120052 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.120183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.120138 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.120183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.120168 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.121949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.121926 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.122102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.122083 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.182807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.182785 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wjr\" (UniqueName: \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\") pod \"simulator-w4-97aa64-kserve-7d57d94fbc-795tm\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.465349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.465262 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:22.843789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:22.843765 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:09:22.845557 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:09:22.845528 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98e43ef4_8e55_4eb6_ab48_41b6d66f9fce.slice/crio-6782f961a978b196d915e2f57b661ad1ab158f0c474421950877dc02735cd2fa WatchSource:0}: Error finding container 6782f961a978b196d915e2f57b661ad1ab158f0c474421950877dc02735cd2fa: Status 404 returned error can't find the container with id 6782f961a978b196d915e2f57b661ad1ab158f0c474421950877dc02735cd2fa Oct 30 22:09:23.110396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:23.110312 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" event={"ID":"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce","Type":"ContainerStarted","Data":"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751"} Oct 30 22:09:23.110396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:23.110361 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" event={"ID":"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce","Type":"ContainerStarted","Data":"6782f961a978b196d915e2f57b661ad1ab158f0c474421950877dc02735cd2fa"} Oct 30 22:09:23.233452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:23.233403 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podStartSLOduration=2.233390221 podStartE2EDuration="2.233390221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:23.230783644 +0000 UTC m=+891.199603209" watchObservedRunningTime="2026-10-30 22:09:23.233390221 +0000 UTC m=+891.202209785" Oct 30 22:09:23.465638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:23.465540 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:26.639470 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:09:26.639435 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:09:32.465463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:32.465431 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:32.479333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:32.479311 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:33.171682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:33.171650 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:09:56.374272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.374241 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:09:56.377808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.377791 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.455694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.455658 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.455820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.455705 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.455880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.455827 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.455880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.455868 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trz7m\" (UniqueName: \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.456121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.456099 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.456159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.456144 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.464267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.464241 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:09:56.485615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.485594 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-075cbb\"/\"kube-root-ca.crt\"" Oct 30 22:09:56.485732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.485683 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-075cbb\"/\"openshift-service-ca.crt\"" Oct 30 22:09:56.485732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.485726 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-075cbb\"/\"premium-w5-075cbb-kserve-self-signed-certs\"" Oct 30 22:09:56.485822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.485730 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-075cbb\"/\"default-dockercfg-dzqtz\"" Oct 30 22:09:56.557088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557099 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trz7m\" (UniqueName: \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557164 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557185 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557215 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557233 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557523 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557553 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.557631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.557618 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.559425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.559405 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.559688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.559670 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.620219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.620190 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trz7m\" (UniqueName: \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\") pod \"premium-w5-075cbb-kserve-69d4b5f686-zqv8j\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:56.639914 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:09:56.639847 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:09:56.689268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:56.689241 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:09:57.071499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:57.071472 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:09:57.072768 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:09:57.072738 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c755482_36fa_474b_a909_a71616315b3c.slice/crio-e68ad46cb22b5cda65a97ae848e24d0954a14e7deec882beede95036d1a1ef15 WatchSource:0}: Error finding container e68ad46cb22b5cda65a97ae848e24d0954a14e7deec882beede95036d1a1ef15: Status 404 returned error can't find the container with id e68ad46cb22b5cda65a97ae848e24d0954a14e7deec882beede95036d1a1ef15 Oct 30 22:09:57.296087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:57.296053 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" event={"ID":"5c755482-36fa-474b-a909-a71616315b3c","Type":"ContainerStarted","Data":"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480"} Oct 30 22:09:57.296087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:57.296097 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" event={"ID":"5c755482-36fa-474b-a909-a71616315b3c","Type":"ContainerStarted","Data":"e68ad46cb22b5cda65a97ae848e24d0954a14e7deec882beede95036d1a1ef15"} Oct 30 22:09:57.418644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:57.418538 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podStartSLOduration=1.418524724 podStartE2EDuration="1.418524724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:57.417736735 +0000 UTC m=+925.386556300" watchObservedRunningTime="2026-10-30 22:09:57.418524724 +0000 UTC m=+925.387344328" Oct 30 22:09:57.689405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:09:57.689331 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:10:06.689369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:06.689324 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:10:06.709159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:06.709131 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:10:07.366812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:07.366784 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:10:26.639835 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:10:26.639798 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:10:36.408017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.407978 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm"] Oct 30 22:10:36.412139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.412117 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.470886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.470853 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm"] Oct 30 22:10:36.475435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.475410 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:36.483005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.482980 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt29j\" (UniqueName: \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.483100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.483074 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.483180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.483154 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.518550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.518523 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:36.518550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.518530 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:10:36.518697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.518627 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:36.584704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.584677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt29j\" (UniqueName: \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.584807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.584790 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.584848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.584814 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.585461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.585435 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.587130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.587110 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.647889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.647858 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt29j\" (UniqueName: \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\") pod \"maas-api-revoke-sub-804277aa-tkwhm\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:36.724517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:36.724441 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:37.101757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:37.101733 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm"] Oct 30 22:10:37.103255 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:10:37.103227 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63603e25_98fc_4282_b5d4_400637c53956.slice/crio-864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b WatchSource:0}: Error finding container 864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b: Status 404 returned error can't find the container with id 864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b Oct 30 22:10:37.515032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:37.514924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" event={"ID":"63603e25-98fc-4282-b5d4-400637c53956","Type":"ContainerStarted","Data":"864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b"} Oct 30 22:10:41.538069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:41.538036 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" event={"ID":"63603e25-98fc-4282-b5d4-400637c53956","Type":"ContainerStarted","Data":"812b3ec16fc6407ba1fc285cdb1ca7c09c5695a6694b136dedde197b611482a7"} Oct 30 22:10:41.665732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:41.665650 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" podStartSLOduration=1.72243039 podStartE2EDuration="5.665636299s" totalImagesPullingTime="3.943205909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:41.664839253 +0000 UTC m=+969.633658818" watchObservedRunningTime="2026-10-30 22:10:41.665636299 +0000 UTC m=+969.634455862" Oct 30 22:10:42.544865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:42.544829 4292 generic.go:356] "Generic (PLEG): container finished" podID="63603e25-98fc-4282-b5d4-400637c53956" containerID="812b3ec16fc6407ba1fc285cdb1ca7c09c5695a6694b136dedde197b611482a7" exitCode=0 Oct 30 22:10:42.545289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:42.544916 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" event={"ID":"63603e25-98fc-4282-b5d4-400637c53956","Type":"ContainerDied","Data":"812b3ec16fc6407ba1fc285cdb1ca7c09c5695a6694b136dedde197b611482a7"} Oct 30 22:10:43.695789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.695765 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:43.760456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.760433 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\" (UniqueName: \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\") pod \"63603e25-98fc-4282-b5d4-400637c53956\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " Oct 30 22:10:43.760610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.760463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\") pod \"63603e25-98fc-4282-b5d4-400637c53956\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " Oct 30 22:10:43.760610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.760531 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\") pod \"63603e25-98fc-4282-b5d4-400637c53956\" (UID: \"63603e25-98fc-4282-b5d4-400637c53956\") " Oct 30 22:10:43.760976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.760953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca" pod "63603e25-98fc-4282-b5d4-400637c53956" (UID: "63603e25-98fc-4282-b5d4-400637c53956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:43.762619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.762593 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca" pod "63603e25-98fc-4282-b5d4-400637c53956" (UID: "63603e25-98fc-4282-b5d4-400637c53956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:43.762619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.762613 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j" pod "63603e25-98fc-4282-b5d4-400637c53956" (UID: "63603e25-98fc-4282-b5d4-400637c53956"). InnerVolumeSpecName "kube-api-access-qt29j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:43.862236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.862178 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt29j\" (UniqueName: \"kubernetes.io/projected/63603e25-98fc-4282-b5d4-400637c53956-kube-api-access-qt29j\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:43.862236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.862202 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63603e25-98fc-4282-b5d4-400637c53956-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:43.862236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:43.862213 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63603e25-98fc-4282-b5d4-400637c53956-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.560208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:44.560165 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" event={"ID":"63603e25-98fc-4282-b5d4-400637c53956","Type":"ContainerDied","Data":"864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b"} Oct 30 22:10:44.560369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:44.560264 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864dbc8562a9d3e5ff1b53352bd769bc2bc2ad4b0f08a285ed5964b8751fad3b" Oct 30 22:10:44.560369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:44.560355 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm" Oct 30 22:10:47.898069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.898026 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:10:47.898594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.898496 4292 cpu_manager.go:414] "containerMap: removing container" podUID="63603e25-98fc-4282-b5d4-400637c53956" containerName="revoke-keys" Oct 30 22:10:47.898594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.898507 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63603e25-98fc-4282-b5d4-400637c53956" containerName="revoke-keys" Oct 30 22:10:47.898736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.898654 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="63603e25-98fc-4282-b5d4-400637c53956" containerName="revoke-keys" Oct 30 22:10:47.902067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.902045 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:47.956758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:47.956729 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-963c3a\"/\"premium-w3-963c3a-kserve-self-signed-certs\"" Oct 30 22:10:48.005986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.005944 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006092 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqj5\" (UniqueName: \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006131 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006242 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.006786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006757 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-963c3a\"/\"openshift-service-ca.crt\"" Oct 30 22:10:48.006928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.006881 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-963c3a\"/\"kube-root-ca.crt\"" Oct 30 22:10:48.011112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.011085 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:10:48.057141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.057107 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-963c3a\"/\"default-dockercfg-54nv5\"" Oct 30 22:10:48.107776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107746 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.107968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107788 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.107968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107840 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.107968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107910 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.107968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107945 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqj5\" (UniqueName: \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.108201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.107979 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.108259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.108205 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.108676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.108331 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.108676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.108634 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.111366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.111336 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.134852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.134827 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.171897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.171822 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqj5\" (UniqueName: \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\") pod \"premium-w3-963c3a-kserve-76999d4f88-7hdxb\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.247541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.247507 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:48.689251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:48.689224 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:10:48.690653 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:10:48.690623 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e077c1_045f_4768_9778_ec5f714b6183.slice/crio-23dd76fa88441e3c8bd7fe9ed8ef664efd4bea7c7ced8f19861d191402eef983 WatchSource:0}: Error finding container 23dd76fa88441e3c8bd7fe9ed8ef664efd4bea7c7ced8f19861d191402eef983: Status 404 returned error can't find the container with id 23dd76fa88441e3c8bd7fe9ed8ef664efd4bea7c7ced8f19861d191402eef983 Oct 30 22:10:49.612040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:49.612006 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" event={"ID":"b3e077c1-045f-4768-9778-ec5f714b6183","Type":"ContainerStarted","Data":"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c"} Oct 30 22:10:49.612432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:49.612051 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" event={"ID":"b3e077c1-045f-4768-9778-ec5f714b6183","Type":"ContainerStarted","Data":"23dd76fa88441e3c8bd7fe9ed8ef664efd4bea7c7ced8f19861d191402eef983"} Oct 30 22:10:49.733063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:49.733010 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podStartSLOduration=2.732996507 podStartE2EDuration="2.732996507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.731968455 +0000 UTC m=+977.700788018" watchObservedRunningTime="2026-10-30 22:10:49.732996507 +0000 UTC m=+977.701816071" Oct 30 22:10:50.248366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:50.248335 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:52.139280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.139247 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c"] Oct 30 22:10:52.142772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.142750 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.202058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.202034 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c"] Oct 30 22:10:52.248506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.248476 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.248699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.248628 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd57\" (UniqueName: \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.248699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.248657 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.249496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.249480 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:52.249568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.249485 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:52.249568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.249531 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:10:52.249669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.249612 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:52.349952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.349929 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.350066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.350023 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd57\" (UniqueName: \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.350066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.350053 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.350605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.350565 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.352274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.352251 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.413395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.413341 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd57\" (UniqueName: \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\") pod \"maas-api-revoke-sub-3a370166-skv8c\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.457300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.457276 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:52.837751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:52.837714 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c"] Oct 30 22:10:52.838712 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:10:52.838683 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec43f881_19ee_42ba_a2d6_886028e7d0a3.slice/crio-6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75 WatchSource:0}: Error finding container 6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75: Status 404 returned error can't find the container with id 6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75 Oct 30 22:10:53.635453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:53.635412 4292 generic.go:356] "Generic (PLEG): container finished" podID="ec43f881-19ee-42ba-a2d6-886028e7d0a3" containerID="90664f32c47218936a20ae384903cb19e6874b349eb7e54256901505817f1af4" exitCode=0 Oct 30 22:10:53.635820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:53.635476 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" event={"ID":"ec43f881-19ee-42ba-a2d6-886028e7d0a3","Type":"ContainerDied","Data":"90664f32c47218936a20ae384903cb19e6874b349eb7e54256901505817f1af4"} Oct 30 22:10:53.635820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:53.635542 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" event={"ID":"ec43f881-19ee-42ba-a2d6-886028e7d0a3","Type":"ContainerStarted","Data":"6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75"} Oct 30 22:10:54.783852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.783829 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:54.874849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.874812 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\") pod \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " Oct 30 22:10:54.874992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.874954 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\" (UniqueName: \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\") pod \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " Oct 30 22:10:54.875071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.875029 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\") pod \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\" (UID: \"ec43f881-19ee-42ba-a2d6-886028e7d0a3\") " Oct 30 22:10:54.875187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.875163 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca" pod "ec43f881-19ee-42ba-a2d6-886028e7d0a3" (UID: "ec43f881-19ee-42ba-a2d6-886028e7d0a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:54.875404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.875384 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.876945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.876915 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57" pod "ec43f881-19ee-42ba-a2d6-886028e7d0a3" (UID: "ec43f881-19ee-42ba-a2d6-886028e7d0a3"). InnerVolumeSpecName "kube-api-access-gjd57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:54.877082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.877059 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca" pod "ec43f881-19ee-42ba-a2d6-886028e7d0a3" (UID: "ec43f881-19ee-42ba-a2d6-886028e7d0a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:54.976652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.976557 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec43f881-19ee-42ba-a2d6-886028e7d0a3-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.976652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:54.976610 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjd57\" (UniqueName: \"kubernetes.io/projected/ec43f881-19ee-42ba-a2d6-886028e7d0a3-kube-api-access-gjd57\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.654114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:55.654085 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" Oct 30 22:10:55.654292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:55.654078 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c" event={"ID":"ec43f881-19ee-42ba-a2d6-886028e7d0a3","Type":"ContainerDied","Data":"6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75"} Oct 30 22:10:55.654292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:55.654220 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1d8526fe55ba796c31a39241f60cbd857b056c88b08a9b688a469a7c29de75" Oct 30 22:10:55.706475 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:10:55.706448 4292 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec43f881_19ee_42ba_a2d6_886028e7d0a3.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:10:56.639073 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:10:56.639034 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:10:58.249022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:58.248989 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:58.261404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:58.261380 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:10:58.680825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:10:58.680798 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:11:05.313937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.313902 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97"] Oct 30 22:11:05.314458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.314442 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ec43f881-19ee-42ba-a2d6-886028e7d0a3" containerName="revoke-keys" Oct 30 22:11:05.314510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.314458 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec43f881-19ee-42ba-a2d6-886028e7d0a3" containerName="revoke-keys" Oct 30 22:11:05.314557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.314548 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec43f881-19ee-42ba-a2d6-886028e7d0a3" containerName="revoke-keys" Oct 30 22:11:05.317994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.317976 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.375074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.375032 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:05.375074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.375030 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:05.375244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.375129 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:05.375293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.375253 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:11:05.377907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.377884 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97"] Oct 30 22:11:05.505717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.505689 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blgf\" (UniqueName: \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.505858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.505727 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.505858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.505823 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.607112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.607041 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blgf\" (UniqueName: \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.607112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.607080 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.607277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.607131 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.607789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.607766 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.609446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.609426 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.670712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.670688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blgf\" (UniqueName: \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\") pod \"maas-api-revoke-sub-d2de42ec-2bg97\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:05.931232 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:05.931159 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:06.311949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:06.311920 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97"] Oct 30 22:11:06.313364 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:06.313337 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1c8dfd_e9b1_4196_972f_38bb21f2479c.slice/crio-c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7 WatchSource:0}: Error finding container c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7: Status 404 returned error can't find the container with id c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7 Oct 30 22:11:06.724564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:06.724530 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" event={"ID":"3e1c8dfd-e9b1-4196-972f-38bb21f2479c","Type":"ContainerStarted","Data":"5a7fdbe879e74518e04eb003520ba7ef4d0a9efa94c12065fe9e04f1cfbadcc7"} Oct 30 22:11:06.725043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:06.724601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" event={"ID":"3e1c8dfd-e9b1-4196-972f-38bb21f2479c","Type":"ContainerStarted","Data":"c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7"} Oct 30 22:11:06.894661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:06.894527 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" podStartSLOduration=1.894511884 podStartE2EDuration="1.894511884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:06.843295482 +0000 UTC m=+994.812115045" watchObservedRunningTime="2026-10-30 22:11:06.894511884 +0000 UTC m=+994.863331447" Oct 30 22:11:07.732035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:07.732001 4292 generic.go:356] "Generic (PLEG): container finished" podID="3e1c8dfd-e9b1-4196-972f-38bb21f2479c" containerID="5a7fdbe879e74518e04eb003520ba7ef4d0a9efa94c12065fe9e04f1cfbadcc7" exitCode=0 Oct 30 22:11:07.732522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:07.732083 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" event={"ID":"3e1c8dfd-e9b1-4196-972f-38bb21f2479c","Type":"ContainerDied","Data":"5a7fdbe879e74518e04eb003520ba7ef4d0a9efa94c12065fe9e04f1cfbadcc7"} Oct 30 22:11:08.063965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.063931 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj"] Oct 30 22:11:08.072651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.070152 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.125210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.125170 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.125360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.125319 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.125478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.125454 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvxr\" (UniqueName: \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.127630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.127596 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj"] Oct 30 22:11:08.226061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.226025 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvxr\" (UniqueName: \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.226229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.226204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.226607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.226560 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.227102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.227078 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.228498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.228476 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.288860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.288831 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvxr\" (UniqueName: \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\") pod \"maas-api-revoke-sub-d1033e88-cf2mj\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.384614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.384545 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:08.764639 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:08.764604 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326fcb1f_29b9_49c7_aec1_d2977b75eeee.slice/crio-361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062 WatchSource:0}: Error finding container 361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062: Status 404 returned error can't find the container with id 361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062 Oct 30 22:11:08.764951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.764843 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj"] Oct 30 22:11:08.874338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.874311 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:08.932047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.932019 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\" (UniqueName: \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\") pod \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " Oct 30 22:11:08.932186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.932065 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\") pod \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " Oct 30 22:11:08.932186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.932095 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\") pod \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\" (UID: \"3e1c8dfd-e9b1-4196-972f-38bb21f2479c\") " Oct 30 22:11:08.932441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.932418 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca" pod "3e1c8dfd-e9b1-4196-972f-38bb21f2479c" (UID: "3e1c8dfd-e9b1-4196-972f-38bb21f2479c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:08.933952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.933931 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf" pod "3e1c8dfd-e9b1-4196-972f-38bb21f2479c" (UID: "3e1c8dfd-e9b1-4196-972f-38bb21f2479c"). InnerVolumeSpecName "kube-api-access-6blgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:08.934016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:08.933991 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca" pod "3e1c8dfd-e9b1-4196-972f-38bb21f2479c" (UID: "3e1c8dfd-e9b1-4196-972f-38bb21f2479c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:09.033502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.033479 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blgf\" (UniqueName: \"kubernetes.io/projected/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-kube-api-access-6blgf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.033502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.033504 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.033666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.033514 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1c8dfd-e9b1-4196-972f-38bb21f2479c-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.745676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.745644 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" Oct 30 22:11:09.745844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.745647 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97" event={"ID":"3e1c8dfd-e9b1-4196-972f-38bb21f2479c","Type":"ContainerDied","Data":"c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7"} Oct 30 22:11:09.745844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.745793 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c60bb638936966ff3f998feed6140068948fffea5399c296207efa90028566c7" Oct 30 22:11:09.747365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.747338 4292 generic.go:356] "Generic (PLEG): container finished" podID="326fcb1f-29b9-49c7-aec1-d2977b75eeee" containerID="2475cf342a982b8b98e24765da2cbd4f2ef3ce5b6fd6f37773fbec7220e9331c" exitCode=0 Oct 30 22:11:09.747475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.747388 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" event={"ID":"326fcb1f-29b9-49c7-aec1-d2977b75eeee","Type":"ContainerDied","Data":"2475cf342a982b8b98e24765da2cbd4f2ef3ce5b6fd6f37773fbec7220e9331c"} Oct 30 22:11:09.747475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:09.747429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" event={"ID":"326fcb1f-29b9-49c7-aec1-d2977b75eeee","Type":"ContainerStarted","Data":"361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062"} Oct 30 22:11:10.899067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.899045 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:10.952246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.952209 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\") pod \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " Oct 30 22:11:10.952387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.952307 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\") pod \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " Oct 30 22:11:10.952387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.952344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\" (UniqueName: \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\") pod \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\" (UID: \"326fcb1f-29b9-49c7-aec1-d2977b75eeee\") " Oct 30 22:11:10.952696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.952662 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca" pod "326fcb1f-29b9-49c7-aec1-d2977b75eeee" (UID: "326fcb1f-29b9-49c7-aec1-d2977b75eeee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:10.953047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.953022 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:10.954362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.954337 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca" pod "326fcb1f-29b9-49c7-aec1-d2977b75eeee" (UID: "326fcb1f-29b9-49c7-aec1-d2977b75eeee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:10.954455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:10.954431 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr" pod "326fcb1f-29b9-49c7-aec1-d2977b75eeee" (UID: "326fcb1f-29b9-49c7-aec1-d2977b75eeee"). InnerVolumeSpecName "kube-api-access-cwvxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:11.053864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:11.053839 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326fcb1f-29b9-49c7-aec1-d2977b75eeee-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:11.053864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:11.053864 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwvxr\" (UniqueName: \"kubernetes.io/projected/326fcb1f-29b9-49c7-aec1-d2977b75eeee-kube-api-access-cwvxr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:11.760362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:11.760325 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" event={"ID":"326fcb1f-29b9-49c7-aec1-d2977b75eeee","Type":"ContainerDied","Data":"361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062"} Oct 30 22:11:11.760362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:11.760349 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj" Oct 30 22:11:11.760563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:11.760378 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361198f08c72462d553109861f6af09b60edb4ebc0bdbc2cd329f24b39ec7062" Oct 30 22:11:25.982186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982148 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd"] Oct 30 22:11:25.982676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982660 4292 cpu_manager.go:414] "containerMap: removing container" podUID="326fcb1f-29b9-49c7-aec1-d2977b75eeee" containerName="revoke-keys" Oct 30 22:11:25.982725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982677 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326fcb1f-29b9-49c7-aec1-d2977b75eeee" containerName="revoke-keys" Oct 30 22:11:25.982725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982691 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3e1c8dfd-e9b1-4196-972f-38bb21f2479c" containerName="revoke-keys" Oct 30 22:11:25.982725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982696 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1c8dfd-e9b1-4196-972f-38bb21f2479c" containerName="revoke-keys" Oct 30 22:11:25.982821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982802 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="326fcb1f-29b9-49c7-aec1-d2977b75eeee" containerName="revoke-keys" Oct 30 22:11:25.982821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.982811 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1c8dfd-e9b1-4196-972f-38bb21f2479c" containerName="revoke-keys" Oct 30 22:11:25.986174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:25.986156 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.045291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.045265 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd"] Oct 30 22:11:26.092090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.092062 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:26.092090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.092087 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:11:26.092286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.092083 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:26.092286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.092168 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:26.101589 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.101556 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5h7\" (UniqueName: \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.101693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.101666 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.101732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.101709 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.202391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.202362 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5h7\" (UniqueName: \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.202667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.202647 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.202754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.202724 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.203205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.203181 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.204936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.204912 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.265813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.265760 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5h7\" (UniqueName: \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\") pod \"maas-api-revoke-sub-8814f4b4-wpptd\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.299586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.299556 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:26.639713 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:11:26.639680 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:11:26.675032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.675004 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd"] Oct 30 22:11:26.676280 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:26.676252 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530d73d6_51b8_40d5_b5a4_f104271bbefe.slice/crio-9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1 WatchSource:0}: Error finding container 9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1: Status 404 returned error can't find the container with id 9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1 Oct 30 22:11:26.849525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:26.849492 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" event={"ID":"530d73d6-51b8-40d5-b5a4-f104271bbefe","Type":"ContainerStarted","Data":"9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1"} Oct 30 22:11:27.858315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:27.858272 4292 generic.go:356] "Generic (PLEG): container finished" podID="530d73d6-51b8-40d5-b5a4-f104271bbefe" containerID="c517750a472b5aada2935167017020ed3d7fa645e1529510b9a9d2496a3446ae" exitCode=0 Oct 30 22:11:27.858732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:27.858351 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" event={"ID":"530d73d6-51b8-40d5-b5a4-f104271bbefe","Type":"ContainerDied","Data":"c517750a472b5aada2935167017020ed3d7fa645e1529510b9a9d2496a3446ae"} Oct 30 22:11:29.018721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.018692 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:29.158848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.158776 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\") pod \"530d73d6-51b8-40d5-b5a4-f104271bbefe\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " Oct 30 22:11:29.159011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.158885 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\") pod \"530d73d6-51b8-40d5-b5a4-f104271bbefe\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " Oct 30 22:11:29.159011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.158934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\" (UniqueName: \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\") pod \"530d73d6-51b8-40d5-b5a4-f104271bbefe\" (UID: \"530d73d6-51b8-40d5-b5a4-f104271bbefe\") " Oct 30 22:11:29.159177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.159150 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca" pod "530d73d6-51b8-40d5-b5a4-f104271bbefe" (UID: "530d73d6-51b8-40d5-b5a4-f104271bbefe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:29.159498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.159475 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:29.161465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.161438 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca" pod "530d73d6-51b8-40d5-b5a4-f104271bbefe" (UID: "530d73d6-51b8-40d5-b5a4-f104271bbefe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:29.161908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.161885 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7" pod "530d73d6-51b8-40d5-b5a4-f104271bbefe" (UID: "530d73d6-51b8-40d5-b5a4-f104271bbefe"). InnerVolumeSpecName "kube-api-access-lf5h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:29.260860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.260835 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530d73d6-51b8-40d5-b5a4-f104271bbefe-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:29.260860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.260861 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5h7\" (UniqueName: \"kubernetes.io/projected/530d73d6-51b8-40d5-b5a4-f104271bbefe-kube-api-access-lf5h7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:29.872300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.872265 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" Oct 30 22:11:29.872484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.872303 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd" event={"ID":"530d73d6-51b8-40d5-b5a4-f104271bbefe","Type":"ContainerDied","Data":"9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1"} Oct 30 22:11:29.872484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:29.872352 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7873ef1405a524f802ad1b7f7880fab314db9d8129b133b08309ba17503fc1" Oct 30 22:11:36.793883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.793848 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv"] Oct 30 22:11:36.794482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.794463 4292 cpu_manager.go:414] "containerMap: removing container" podUID="530d73d6-51b8-40d5-b5a4-f104271bbefe" containerName="revoke-keys" Oct 30 22:11:36.794529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.794483 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530d73d6-51b8-40d5-b5a4-f104271bbefe" containerName="revoke-keys" Oct 30 22:11:36.794596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.794587 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="530d73d6-51b8-40d5-b5a4-f104271bbefe" containerName="revoke-keys" Oct 30 22:11:36.798186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.798163 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.840722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.840693 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.840837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.840756 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.840895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.840875 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tlw\" (UniqueName: \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.856820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.856793 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv"] Oct 30 22:11:36.904180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.904149 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:36.904180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.904152 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:11:36.904180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.904152 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:36.904390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.904237 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:36.942088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.942060 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.942235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.942132 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tlw\" (UniqueName: \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.942235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.942210 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.942789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.942762 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:36.944399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:36.944369 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:37.005125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:37.005099 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tlw\" (UniqueName: \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\") pod \"maas-api-revoke-sub-07c66426-hflpv\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:37.111794 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:37.111771 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:37.488937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:37.488906 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv"] Oct 30 22:11:37.490154 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:37.490126 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d704afd_e3e9_4f8d_817a_673b6d2c6f2c.slice/crio-fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300 WatchSource:0}: Error finding container fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300: Status 404 returned error can't find the container with id fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300 Oct 30 22:11:37.920569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:37.920539 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" event={"ID":"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c","Type":"ContainerStarted","Data":"fc625de954b3bb49e7b1992f49e1f5abd75be8b10bfd488ac965d27118ac0738"} Oct 30 22:11:37.920920 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:37.920598 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" event={"ID":"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c","Type":"ContainerStarted","Data":"fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300"} Oct 30 22:11:38.045080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:38.045033 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" podStartSLOduration=2.0450203240000002 podStartE2EDuration="2.045020324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:38.044742911 +0000 UTC m=+1026.013562476" watchObservedRunningTime="2026-10-30 22:11:38.045020324 +0000 UTC m=+1026.013839917" Oct 30 22:11:38.927186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:38.927153 4292 generic.go:356] "Generic (PLEG): container finished" podID="0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" containerID="fc625de954b3bb49e7b1992f49e1f5abd75be8b10bfd488ac965d27118ac0738" exitCode=0 Oct 30 22:11:38.927642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:38.927194 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" event={"ID":"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c","Type":"ContainerDied","Data":"fc625de954b3bb49e7b1992f49e1f5abd75be8b10bfd488ac965d27118ac0738"} Oct 30 22:11:40.068055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.068029 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:40.175239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.175208 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\" (UniqueName: \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\") pod \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " Oct 30 22:11:40.175366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.175275 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\") pod \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " Oct 30 22:11:40.175366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.175357 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\") pod \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\" (UID: \"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c\") " Oct 30 22:11:40.175620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.175563 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca" pod "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" (UID: "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:40.175786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.175771 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:40.177248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.177229 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw" pod "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" (UID: "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c"). InnerVolumeSpecName "kube-api-access-g8tlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:40.177311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.177290 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca" pod "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" (UID: "0d704afd-e3e9-4f8d-817a-673b6d2c6f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:40.276372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.276342 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tlw\" (UniqueName: \"kubernetes.io/projected/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-kube-api-access-g8tlw\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:40.276372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.276369 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:40.939024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.938994 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" Oct 30 22:11:40.939192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.939023 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv" event={"ID":"0d704afd-e3e9-4f8d-817a-673b6d2c6f2c","Type":"ContainerDied","Data":"fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300"} Oct 30 22:11:40.939192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:40.939069 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa80dfc30248e09b0f64ceeb26c5565d44a48e6267349c4af38c091c776ff300" Oct 30 22:11:51.168644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.168553 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm"] Oct 30 22:11:51.169073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.169059 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" containerName="revoke-keys" Oct 30 22:11:51.169120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.169074 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" containerName="revoke-keys" Oct 30 22:11:51.169173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.169165 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" containerName="revoke-keys" Oct 30 22:11:51.172361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.172344 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.233311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.233285 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm"] Oct 30 22:11:51.279520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.279492 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:51.279680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.279535 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:51.279680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.279611 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:51.280790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.280771 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:11:51.290242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.290222 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.290311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.290256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.290367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.290341 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5dnr\" (UniqueName: \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.391160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.391134 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.391290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.391183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.391290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.391240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5dnr\" (UniqueName: \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.391860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.391840 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.393472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.393449 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.453980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.453928 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5dnr\" (UniqueName: \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\") pod \"maas-api-revoke-sub-9a2cfac8-9z5vm\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.485369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.485348 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:51.863346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:51.863324 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm"] Oct 30 22:11:51.864292 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:51.864264 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135c6e87_cb11_4fce_9901_26f5a01fc5f4.slice/crio-794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6 WatchSource:0}: Error finding container 794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6: Status 404 returned error can't find the container with id 794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6 Oct 30 22:11:52.001242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:52.001207 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" event={"ID":"135c6e87-cb11-4fce-9901-26f5a01fc5f4","Type":"ContainerStarted","Data":"794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6"} Oct 30 22:11:53.008088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:53.008005 4292 generic.go:356] "Generic (PLEG): container finished" podID="135c6e87-cb11-4fce-9901-26f5a01fc5f4" containerID="c3d90df04ecda4a3220e1fa0f8c1d18225c2e303c985118ff521e9e23b43eaf3" exitCode=0 Oct 30 22:11:53.008088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:53.008041 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" event={"ID":"135c6e87-cb11-4fce-9901-26f5a01fc5f4","Type":"ContainerDied","Data":"c3d90df04ecda4a3220e1fa0f8c1d18225c2e303c985118ff521e9e23b43eaf3"} Oct 30 22:11:54.162677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.162650 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:54.324481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.324441 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\") pod \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " Oct 30 22:11:54.324693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.324559 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\") pod \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " Oct 30 22:11:54.324693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.324611 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\" (UniqueName: \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\") pod \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\" (UID: \"135c6e87-cb11-4fce-9901-26f5a01fc5f4\") " Oct 30 22:11:54.324897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.324868 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca" pod "135c6e87-cb11-4fce-9901-26f5a01fc5f4" (UID: "135c6e87-cb11-4fce-9901-26f5a01fc5f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:54.325133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.325116 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:54.326754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.326723 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr" pod "135c6e87-cb11-4fce-9901-26f5a01fc5f4" (UID: "135c6e87-cb11-4fce-9901-26f5a01fc5f4"). InnerVolumeSpecName "kube-api-access-j5dnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:54.327068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.327039 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca" pod "135c6e87-cb11-4fce-9901-26f5a01fc5f4" (UID: "135c6e87-cb11-4fce-9901-26f5a01fc5f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:54.426286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.426249 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c6e87-cb11-4fce-9901-26f5a01fc5f4-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:54.426286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:54.426284 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5dnr\" (UniqueName: \"kubernetes.io/projected/135c6e87-cb11-4fce-9901-26f5a01fc5f4-kube-api-access-j5dnr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:55.030930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.030866 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" Oct 30 22:11:55.031145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.030855 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm" event={"ID":"135c6e87-cb11-4fce-9901-26f5a01fc5f4","Type":"ContainerDied","Data":"794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6"} Oct 30 22:11:55.031341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.031311 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794bd620b72c4fd10ab480b7bac2bc9c0f149b79beb088e15440ef97a26539e6" Oct 30 22:11:55.292206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.292118 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4"] Oct 30 22:11:55.293587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.293553 4292 cpu_manager.go:414] "containerMap: removing container" podUID="135c6e87-cb11-4fce-9901-26f5a01fc5f4" containerName="revoke-keys" Oct 30 22:11:55.293688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.293594 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135c6e87-cb11-4fce-9901-26f5a01fc5f4" containerName="revoke-keys" Oct 30 22:11:55.293749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.293736 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="135c6e87-cb11-4fce-9901-26f5a01fc5f4" containerName="revoke-keys" Oct 30 22:11:55.297699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.297680 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.353610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.352740 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:55.355635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.355602 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4"] Oct 30 22:11:55.358417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.358394 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:55.358520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.358396 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:55.358520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.358453 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:11:55.445380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.445334 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4tq\" (UniqueName: \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.445543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.445442 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.445652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.445585 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.546357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.546331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.546476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.546428 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4tq\" (UniqueName: \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.546518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.546498 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.547112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.547087 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.548662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.548643 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.609870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.609843 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4tq\" (UniqueName: \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\") pod \"maas-api-revoke-sub-23cd8b01-tqtn4\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.615663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.615645 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:55.991852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:55.991816 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4"] Oct 30 22:11:55.992977 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:11:55.992948 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4a4333_3032_4de8_a516_fca6b5ea9719.slice/crio-bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee WatchSource:0}: Error finding container bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee: Status 404 returned error can't find the container with id bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee Oct 30 22:11:56.037585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:56.037546 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" event={"ID":"fb4a4333-3032-4de8-a516-fca6b5ea9719","Type":"ContainerStarted","Data":"bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee"} Oct 30 22:11:56.638868 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:11:56.638827 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:11:57.044283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:57.044202 4292 generic.go:356] "Generic (PLEG): container finished" podID="fb4a4333-3032-4de8-a516-fca6b5ea9719" containerID="f4235624a0d0122937771026c16092f1847a791787b7525a99d12cebad9ca2dd" exitCode=0 Oct 30 22:11:57.044283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:57.044255 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" event={"ID":"fb4a4333-3032-4de8-a516-fca6b5ea9719","Type":"ContainerDied","Data":"f4235624a0d0122937771026c16092f1847a791787b7525a99d12cebad9ca2dd"} Oct 30 22:11:58.200076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.200047 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:58.274187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.274152 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\") pod \"fb4a4333-3032-4de8-a516-fca6b5ea9719\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " Oct 30 22:11:58.274352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.274286 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\" (UniqueName: \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\") pod \"fb4a4333-3032-4de8-a516-fca6b5ea9719\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " Oct 30 22:11:58.274352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.274324 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\") pod \"fb4a4333-3032-4de8-a516-fca6b5ea9719\" (UID: \"fb4a4333-3032-4de8-a516-fca6b5ea9719\") " Oct 30 22:11:58.274759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.274729 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca" pod "fb4a4333-3032-4de8-a516-fca6b5ea9719" (UID: "fb4a4333-3032-4de8-a516-fca6b5ea9719"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:58.274974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.274941 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.276307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.276285 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca" pod "fb4a4333-3032-4de8-a516-fca6b5ea9719" (UID: "fb4a4333-3032-4de8-a516-fca6b5ea9719"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:58.276431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.276409 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq" pod "fb4a4333-3032-4de8-a516-fca6b5ea9719" (UID: "fb4a4333-3032-4de8-a516-fca6b5ea9719"). InnerVolumeSpecName "kube-api-access-8f4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:58.375920 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.375831 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f4tq\" (UniqueName: \"kubernetes.io/projected/fb4a4333-3032-4de8-a516-fca6b5ea9719-kube-api-access-8f4tq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.375920 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:58.375875 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4a4333-3032-4de8-a516-fca6b5ea9719-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.056707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:59.056675 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" Oct 30 22:11:59.056707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:59.056686 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4" event={"ID":"fb4a4333-3032-4de8-a516-fca6b5ea9719","Type":"ContainerDied","Data":"bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee"} Oct 30 22:11:59.056935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:11:59.056733 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0b49284789ba16caa3ef17e85956a0feac317fad20efdc3662583fadbe34ee" Oct 30 22:12:03.144492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.144460 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h"] Oct 30 22:12:03.145014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.144999 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fb4a4333-3032-4de8-a516-fca6b5ea9719" containerName="revoke-keys" Oct 30 22:12:03.145052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.145015 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4a4333-3032-4de8-a516-fca6b5ea9719" containerName="revoke-keys" Oct 30 22:12:03.145097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.145088 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4a4333-3032-4de8-a516-fca6b5ea9719" containerName="revoke-keys" Oct 30 22:12:03.148431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.148415 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.202999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.202974 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:03.224172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.224140 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.224282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.224202 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.224361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.224336 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqp6d\" (UniqueName: \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.252850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.252829 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:03.253051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.253035 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:12:03.253098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.253034 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:03.256024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.256006 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h"] Oct 30 22:12:03.325278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.325253 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.325388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.325310 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqp6d\" (UniqueName: \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.325425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.325411 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.326017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.325997 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.327701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.327682 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.388630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.388602 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqp6d\" (UniqueName: \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\") pod \"maas-api-revoke-sub-5eadaf90-tvl2h\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.460922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.460874 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:03.837686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:03.837664 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h"] Oct 30 22:12:03.839627 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:03.839597 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4514ae_7b5f_4382_a3a4_ce759d68b05c.slice/crio-f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578 WatchSource:0}: Error finding container f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578: Status 404 returned error can't find the container with id f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578 Oct 30 22:12:04.085976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:04.085944 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" event={"ID":"da4514ae-7b5f-4382-a3a4-ce759d68b05c","Type":"ContainerStarted","Data":"f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578"} Oct 30 22:12:05.094412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:05.094380 4292 generic.go:356] "Generic (PLEG): container finished" podID="da4514ae-7b5f-4382-a3a4-ce759d68b05c" containerID="61e8c27e8fb82a945f372f44613c27a9efa0cace06cffc51df70ac677f3445bb" exitCode=0 Oct 30 22:12:05.094833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:05.094429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" event={"ID":"da4514ae-7b5f-4382-a3a4-ce759d68b05c","Type":"ContainerDied","Data":"61e8c27e8fb82a945f372f44613c27a9efa0cace06cffc51df70ac677f3445bb"} Oct 30 22:12:06.341896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.341865 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:06.476019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.475927 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\") pod \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " Oct 30 22:12:06.476177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.476094 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\") pod \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " Oct 30 22:12:06.476177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.476125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\" (UniqueName: \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\") pod \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\" (UID: \"da4514ae-7b5f-4382-a3a4-ce759d68b05c\") " Oct 30 22:12:06.476465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.476431 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca" pod "da4514ae-7b5f-4382-a3a4-ce759d68b05c" (UID: "da4514ae-7b5f-4382-a3a4-ce759d68b05c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:06.478322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.478292 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d" pod "da4514ae-7b5f-4382-a3a4-ce759d68b05c" (UID: "da4514ae-7b5f-4382-a3a4-ce759d68b05c"). InnerVolumeSpecName "kube-api-access-bqp6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:06.478839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.478814 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca" pod "da4514ae-7b5f-4382-a3a4-ce759d68b05c" (UID: "da4514ae-7b5f-4382-a3a4-ce759d68b05c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:06.579442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.579400 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:06.579711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.579689 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqp6d\" (UniqueName: \"kubernetes.io/projected/da4514ae-7b5f-4382-a3a4-ce759d68b05c-kube-api-access-bqp6d\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:06.579828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:06.579811 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4514ae-7b5f-4382-a3a4-ce759d68b05c-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:07.109378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:07.109350 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" Oct 30 22:12:07.109567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:07.109320 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h" event={"ID":"da4514ae-7b5f-4382-a3a4-ce759d68b05c","Type":"ContainerDied","Data":"f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578"} Oct 30 22:12:07.109567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:07.109504 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f225a0f4c4c882132be8de3746a11eb73ea7e067fd780c8b4431280689493578" Oct 30 22:12:11.331533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.331498 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx"] Oct 30 22:12:11.332029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.332015 4292 cpu_manager.go:414] "containerMap: removing container" podUID="da4514ae-7b5f-4382-a3a4-ce759d68b05c" containerName="revoke-keys" Oct 30 22:12:11.332074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.332030 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4514ae-7b5f-4382-a3a4-ce759d68b05c" containerName="revoke-keys" Oct 30 22:12:11.332172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.332162 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4514ae-7b5f-4382-a3a4-ce759d68b05c" containerName="revoke-keys" Oct 30 22:12:11.337149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.337130 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.395408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.395373 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx"] Oct 30 22:12:11.441438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.441415 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:11.441731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.441711 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:11.441798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.441718 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:12:11.441798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.441749 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:11.445298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.445279 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rck\" (UniqueName: \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.445406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.445392 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.445443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.445430 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.546556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.546529 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.546706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.546565 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.546706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.546651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rck\" (UniqueName: \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.547197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.547175 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.548928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.548910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.610099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.610041 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rck\" (UniqueName: \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\") pod \"maas-api-revoke-sub-93dc2fc1-sxscx\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.650251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.650227 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:11.916850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.916782 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5"] Oct 30 22:12:11.923648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.923617 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:11.980187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:11.980163 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5"] Oct 30 22:12:12.028757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.028727 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx"] Oct 30 22:12:12.029668 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:12.029639 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda257b4d5_4829_485d_a0c1_5643772e4c23.slice/crio-04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5 WatchSource:0}: Error finding container 04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5: Status 404 returned error can't find the container with id 04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5 Oct 30 22:12:12.052686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.052663 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.052777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.052699 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vh89\" (UniqueName: \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.052886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.052863 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.145482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.145455 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" event={"ID":"a257b4d5-4829-485d-a0c1-5643772e4c23","Type":"ContainerStarted","Data":"04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5"} Oct 30 22:12:12.153756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.153737 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.153832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.153804 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.153832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.153828 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh89\" (UniqueName: \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.154438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.154419 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.155960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.155943 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.217113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.217088 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh89\" (UniqueName: \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\") pod \"maas-api-revoke-sub-12e0de7f-j99r5\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.239513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.239491 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:12.624551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:12.624523 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5"] Oct 30 22:12:12.625265 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:12.625241 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd688ff4b_1b4b_4172_b542_c1bd6e06c179.slice/crio-50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9 WatchSource:0}: Error finding container 50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9: Status 404 returned error can't find the container with id 50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9 Oct 30 22:12:13.147332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.147304 4292 generic.go:356] "Generic (PLEG): container finished" podID="a257b4d5-4829-485d-a0c1-5643772e4c23" containerID="848feba548c63fe53dd46ec9162c8d5b376843df27e70457d5858a6a3106eb1d" exitCode=0 Oct 30 22:12:13.147488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.147395 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" event={"ID":"a257b4d5-4829-485d-a0c1-5643772e4c23","Type":"ContainerDied","Data":"848feba548c63fe53dd46ec9162c8d5b376843df27e70457d5858a6a3106eb1d"} Oct 30 22:12:13.148976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.148947 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" event={"ID":"d688ff4b-1b4b-4172-b542-c1bd6e06c179","Type":"ContainerStarted","Data":"b8c97452d4a0c205d64520a583451497157136f7d5b45bf20f6f8c6a330dc60d"} Oct 30 22:12:13.149080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.148992 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" event={"ID":"d688ff4b-1b4b-4172-b542-c1bd6e06c179","Type":"ContainerStarted","Data":"50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9"} Oct 30 22:12:13.389176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.389123 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" podStartSLOduration=2.389109372 podStartE2EDuration="2.389109372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:13.38624507 +0000 UTC m=+1061.355064636" watchObservedRunningTime="2026-10-30 22:12:13.389109372 +0000 UTC m=+1061.357928936" Oct 30 22:12:13.899865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.899835 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq"] Oct 30 22:12:13.903681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.903659 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:13.963197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.963168 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq"] Oct 30 22:12:13.972806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.972780 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:13.972907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.972880 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgql6\" (UniqueName: \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:13.973070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:13.973049 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.073651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.073615 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.073782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.073673 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.073782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.073753 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgql6\" (UniqueName: \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.074301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.074280 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.075960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.075941 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.137038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.137010 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgql6\" (UniqueName: \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\") pod \"maas-api-revoke-sub-8800e8c4-x96bq\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.155751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.155689 4292 generic.go:356] "Generic (PLEG): container finished" podID="d688ff4b-1b4b-4172-b542-c1bd6e06c179" containerID="b8c97452d4a0c205d64520a583451497157136f7d5b45bf20f6f8c6a330dc60d" exitCode=0 Oct 30 22:12:14.155751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.155730 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" event={"ID":"d688ff4b-1b4b-4172-b542-c1bd6e06c179","Type":"ContainerDied","Data":"b8c97452d4a0c205d64520a583451497157136f7d5b45bf20f6f8c6a330dc60d"} Oct 30 22:12:14.216732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.216696 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:14.307331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.307308 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:14.376111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.376083 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\") pod \"a257b4d5-4829-485d-a0c1-5643772e4c23\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " Oct 30 22:12:14.376268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.376229 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\" (UniqueName: \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\") pod \"a257b4d5-4829-485d-a0c1-5643772e4c23\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " Oct 30 22:12:14.376268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.376251 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\") pod \"a257b4d5-4829-485d-a0c1-5643772e4c23\" (UID: \"a257b4d5-4829-485d-a0c1-5643772e4c23\") " Oct 30 22:12:14.376768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.376735 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca" pod "a257b4d5-4829-485d-a0c1-5643772e4c23" (UID: "a257b4d5-4829-485d-a0c1-5643772e4c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:14.378106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.378079 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca" pod "a257b4d5-4829-485d-a0c1-5643772e4c23" (UID: "a257b4d5-4829-485d-a0c1-5643772e4c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:14.378398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.378379 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck" pod "a257b4d5-4829-485d-a0c1-5643772e4c23" (UID: "a257b4d5-4829-485d-a0c1-5643772e4c23"). InnerVolumeSpecName "kube-api-access-75rck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:14.477880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.477815 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:14.477880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.477840 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rck\" (UniqueName: \"kubernetes.io/projected/a257b4d5-4829-485d-a0c1-5643772e4c23-kube-api-access-75rck\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:14.477880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.477852 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a257b4d5-4829-485d-a0c1-5643772e4c23-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:14.606129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:14.606098 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq"] Oct 30 22:12:14.606389 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:14.606362 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b6b58d_910f_4ebf_9632_acfcb35e667a.slice/crio-de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373 WatchSource:0}: Error finding container de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373: Status 404 returned error can't find the container with id de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373 Oct 30 22:12:15.143535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.143462 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8"] Oct 30 22:12:15.143997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.143984 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a257b4d5-4829-485d-a0c1-5643772e4c23" containerName="revoke-keys" Oct 30 22:12:15.144040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.143998 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a257b4d5-4829-485d-a0c1-5643772e4c23" containerName="revoke-keys" Oct 30 22:12:15.144087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.144078 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a257b4d5-4829-485d-a0c1-5643772e4c23" containerName="revoke-keys" Oct 30 22:12:15.147340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.147324 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.163845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.163826 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" Oct 30 22:12:15.163950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.163868 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx" event={"ID":"a257b4d5-4829-485d-a0c1-5643772e4c23","Type":"ContainerDied","Data":"04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5"} Oct 30 22:12:15.163950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.163926 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04af9815f0a9c1e014393467a243e98f013264b1c74514cab3615af81f2284d5" Oct 30 22:12:15.165623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.165599 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" event={"ID":"96b6b58d-910f-4ebf-9632-acfcb35e667a","Type":"ContainerStarted","Data":"fb2ecdaf420cdbb567e7f309cc6cefead413c11ccab0a00fb61babbcdb1ba287"} Oct 30 22:12:15.165729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.165647 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" event={"ID":"96b6b58d-910f-4ebf-9632-acfcb35e667a","Type":"ContainerStarted","Data":"de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373"} Oct 30 22:12:15.185565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.185538 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.185833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.185811 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv84q\" (UniqueName: \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.185937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.185865 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.207200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.207170 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8"] Oct 30 22:12:15.287627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.287585 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.287796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.287703 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv84q\" (UniqueName: \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.287796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.287754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.288377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.288353 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.289828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.289808 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.326248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.326228 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:15.352057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.352036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv84q\" (UniqueName: \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\") pod \"maas-api-revoke-sub-49efaede-gpgk8\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.388474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.388451 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\") pod \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " Oct 30 22:12:15.388564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.388506 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\" (UniqueName: \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\") pod \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " Oct 30 22:12:15.388725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.388711 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\") pod \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\" (UID: \"d688ff4b-1b4b-4172-b542-c1bd6e06c179\") " Oct 30 22:12:15.389042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.389019 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca" pod "d688ff4b-1b4b-4172-b542-c1bd6e06c179" (UID: "d688ff4b-1b4b-4172-b542-c1bd6e06c179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:15.390342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.390322 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca" pod "d688ff4b-1b4b-4172-b542-c1bd6e06c179" (UID: "d688ff4b-1b4b-4172-b542-c1bd6e06c179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:15.390785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.390762 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89" pod "d688ff4b-1b4b-4172-b542-c1bd6e06c179" (UID: "d688ff4b-1b4b-4172-b542-c1bd6e06c179"). InnerVolumeSpecName "kube-api-access-2vh89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:15.447054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.446984 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" podStartSLOduration=2.44697429 podStartE2EDuration="2.44697429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:15.442839716 +0000 UTC m=+1063.411659281" watchObservedRunningTime="2026-10-30 22:12:15.44697429 +0000 UTC m=+1063.415793853" Oct 30 22:12:15.460850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.460829 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:15.490508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.490483 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:15.490508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.490510 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d688ff4b-1b4b-4172-b542-c1bd6e06c179-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:15.490660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.490520 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vh89\" (UniqueName: \"kubernetes.io/projected/d688ff4b-1b4b-4172-b542-c1bd6e06c179-kube-api-access-2vh89\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:15.843639 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:15.843608 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdb1fc5_45a9_4f6d_ab34_078c88e59025.slice/crio-2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682 WatchSource:0}: Error finding container 2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682: Status 404 returned error can't find the container with id 2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682 Oct 30 22:12:15.843940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:15.843915 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8"] Oct 30 22:12:16.172636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.172604 4292 generic.go:356] "Generic (PLEG): container finished" podID="96b6b58d-910f-4ebf-9632-acfcb35e667a" containerID="fb2ecdaf420cdbb567e7f309cc6cefead413c11ccab0a00fb61babbcdb1ba287" exitCode=0 Oct 30 22:12:16.173006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.172636 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" event={"ID":"96b6b58d-910f-4ebf-9632-acfcb35e667a","Type":"ContainerDied","Data":"fb2ecdaf420cdbb567e7f309cc6cefead413c11ccab0a00fb61babbcdb1ba287"} Oct 30 22:12:16.174114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.174098 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" Oct 30 22:12:16.174210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.174137 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5" event={"ID":"d688ff4b-1b4b-4172-b542-c1bd6e06c179","Type":"ContainerDied","Data":"50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9"} Oct 30 22:12:16.174210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.174179 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50000e93f71ed4b7ba82de6c91a33b04f80271801826c868ec6fe39c2adb4eb9" Oct 30 22:12:16.175731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.175704 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" event={"ID":"ffdb1fc5-45a9-4f6d-ab34-078c88e59025","Type":"ContainerStarted","Data":"bf8a6523e849cad01c801010df0fafadcc00a57812fe7bb3d02e5c5ec112b6b3"} Oct 30 22:12:16.175834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.175746 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" event={"ID":"ffdb1fc5-45a9-4f6d-ab34-078c88e59025","Type":"ContainerStarted","Data":"2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682"} Oct 30 22:12:16.194170 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:12:16.194150 4292 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd688ff4b_1b4b_4172_b542_c1bd6e06c179.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:12:16.423089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.422980 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" podStartSLOduration=1.422965906 podStartE2EDuration="1.422965906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:16.42110665 +0000 UTC m=+1064.389926226" watchObservedRunningTime="2026-10-30 22:12:16.422965906 +0000 UTC m=+1064.391785470" Oct 30 22:12:16.720564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.720494 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:12:16.721046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.721032 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d688ff4b-1b4b-4172-b542-c1bd6e06c179" containerName="revoke-keys" Oct 30 22:12:16.721088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.721047 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d688ff4b-1b4b-4172-b542-c1bd6e06c179" containerName="revoke-keys" Oct 30 22:12:16.721141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.721131 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d688ff4b-1b4b-4172-b542-c1bd6e06c179" containerName="revoke-keys" Oct 30 22:12:16.724367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.724348 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.787316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.787285 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:12:16.809724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.809686 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.809724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.809724 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.809961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.809869 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.809983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.810032 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.810081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.810152 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.810175 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5q6\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.810299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.810207 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911015 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911176 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911217 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5q6\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911258 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911289 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911380 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911454 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911478 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911520 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911687 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.911868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.911837 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.912158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.912137 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.913515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.913493 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.913909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.913887 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.974410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.974326 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:16.974539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:16.974441 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5q6\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:17.036868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.036836 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:17.182831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.182793 4292 generic.go:356] "Generic (PLEG): container finished" podID="ffdb1fc5-45a9-4f6d-ab34-078c88e59025" containerID="bf8a6523e849cad01c801010df0fafadcc00a57812fe7bb3d02e5c5ec112b6b3" exitCode=0 Oct 30 22:12:17.183281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.182879 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" event={"ID":"ffdb1fc5-45a9-4f6d-ab34-078c88e59025","Type":"ContainerDied","Data":"bf8a6523e849cad01c801010df0fafadcc00a57812fe7bb3d02e5c5ec112b6b3"} Oct 30 22:12:17.313584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.313542 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:17.417024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.416991 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\" (UniqueName: \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\") pod \"96b6b58d-910f-4ebf-9632-acfcb35e667a\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " Oct 30 22:12:17.417183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.417038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\") pod \"96b6b58d-910f-4ebf-9632-acfcb35e667a\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " Oct 30 22:12:17.417183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.417167 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\") pod \"96b6b58d-910f-4ebf-9632-acfcb35e667a\" (UID: \"96b6b58d-910f-4ebf-9632-acfcb35e667a\") " Oct 30 22:12:17.417342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.417328 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca" pod "96b6b58d-910f-4ebf-9632-acfcb35e667a" (UID: "96b6b58d-910f-4ebf-9632-acfcb35e667a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:17.417627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.417598 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:17.418942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.418922 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6" pod "96b6b58d-910f-4ebf-9632-acfcb35e667a" (UID: "96b6b58d-910f-4ebf-9632-acfcb35e667a"). InnerVolumeSpecName "kube-api-access-vgql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:17.419015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.418969 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca" pod "96b6b58d-910f-4ebf-9632-acfcb35e667a" (UID: "96b6b58d-910f-4ebf-9632-acfcb35e667a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:17.474138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.474106 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:12:17.476096 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:17.476073 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc8ef88_62d4_42af_9726_09c6d854955c.slice/crio-e671f6d89b968dc494d936e6b294edf79a5ae3400667450d0a31c32e3712a050 WatchSource:0}: Error finding container e671f6d89b968dc494d936e6b294edf79a5ae3400667450d0a31c32e3712a050: Status 404 returned error can't find the container with id e671f6d89b968dc494d936e6b294edf79a5ae3400667450d0a31c32e3712a050 Oct 30 22:12:17.479088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.479054 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:17.479175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.479128 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:17.479175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.479163 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:17.518862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.518839 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgql6\" (UniqueName: \"kubernetes.io/projected/96b6b58d-910f-4ebf-9632-acfcb35e667a-kube-api-access-vgql6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:17.518862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.518862 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b58d-910f-4ebf-9632-acfcb35e667a-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:17.780401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.780332 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:12:17.780880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.780864 4292 cpu_manager.go:414] "containerMap: removing container" podUID="96b6b58d-910f-4ebf-9632-acfcb35e667a" containerName="revoke-keys" Oct 30 22:12:17.780941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.780881 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b6b58d-910f-4ebf-9632-acfcb35e667a" containerName="revoke-keys" Oct 30 22:12:17.780990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.780980 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b6b58d-910f-4ebf-9632-acfcb35e667a" containerName="revoke-keys" Oct 30 22:12:17.784540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.784518 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829496 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829555 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf8s\" (UniqueName: \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829613 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829646 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829725 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.829834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.829817 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.841027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.841000 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-77cb1f\"/\"kube-root-ca.crt\"" Oct 30 22:12:17.841274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.841234 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-77cb1f\"/\"openshift-service-ca.crt\"" Oct 30 22:12:17.841393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.841349 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-77cb1f\"/\"default-dockercfg-4sfbv\"" Oct 30 22:12:17.841497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.841480 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-77cb1f\"/\"distinct-w1-77cb1f-kserve-self-signed-certs\"" Oct 30 22:12:17.844132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.844105 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:12:17.934931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.934829 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935528 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935664 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935790 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935814 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf8s\" (UniqueName: \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.935873 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.936538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.936166 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.937804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.937777 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.940879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.940849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:17.941140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:17.941117 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:18.000197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.000157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf8s\" (UniqueName: \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\") pod \"distinct-w1-77cb1f-kserve-5996685c7b-nmqqf\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:18.102098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.101880 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:18.225203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.225166 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" event={"ID":"efc8ef88-62d4-42af-9726-09c6d854955c","Type":"ContainerStarted","Data":"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72"} Oct 30 22:12:18.225655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.225221 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" event={"ID":"efc8ef88-62d4-42af-9726-09c6d854955c","Type":"ContainerStarted","Data":"e671f6d89b968dc494d936e6b294edf79a5ae3400667450d0a31c32e3712a050"} Oct 30 22:12:18.227773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.227749 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" Oct 30 22:12:18.228135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.228117 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq" event={"ID":"96b6b58d-910f-4ebf-9632-acfcb35e667a","Type":"ContainerDied","Data":"de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373"} Oct 30 22:12:18.228229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.228150 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de975ab88c5064f4adbc92ea0ce5720058ae5383dc691a8950ac689d0113a373" Oct 30 22:12:18.361189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.360408 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podStartSLOduration=2.360386658 podStartE2EDuration="2.360386658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:18.357060259 +0000 UTC m=+1066.325879848" watchObservedRunningTime="2026-10-30 22:12:18.360386658 +0000 UTC m=+1066.329206223" Oct 30 22:12:18.426348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.426320 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:18.520548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.520522 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:12:18.521154 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:18.521127 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod690fa1e2_df19_4660_a8ae_f476f18a61f7.slice/crio-8bce7e67172bbcec138574a97ddf35d114f5f3c9e9aef990df421259fb3772e9 WatchSource:0}: Error finding container 8bce7e67172bbcec138574a97ddf35d114f5f3c9e9aef990df421259fb3772e9: Status 404 returned error can't find the container with id 8bce7e67172bbcec138574a97ddf35d114f5f3c9e9aef990df421259fb3772e9 Oct 30 22:12:18.544391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.544362 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\" (UniqueName: \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\") pod \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " Oct 30 22:12:18.544527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.544516 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\") pod \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " Oct 30 22:12:18.544635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.544613 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\") pod \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\" (UID: \"ffdb1fc5-45a9-4f6d-ab34-078c88e59025\") " Oct 30 22:12:18.545006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.544978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca" pod "ffdb1fc5-45a9-4f6d-ab34-078c88e59025" (UID: "ffdb1fc5-45a9-4f6d-ab34-078c88e59025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:18.545166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.545086 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:18.547184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.547151 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q" pod "ffdb1fc5-45a9-4f6d-ab34-078c88e59025" (UID: "ffdb1fc5-45a9-4f6d-ab34-078c88e59025"). InnerVolumeSpecName "kube-api-access-xv84q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:18.547828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.547799 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca" pod "ffdb1fc5-45a9-4f6d-ab34-078c88e59025" (UID: "ffdb1fc5-45a9-4f6d-ab34-078c88e59025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:18.651077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.648456 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv84q\" (UniqueName: \"kubernetes.io/projected/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-kube-api-access-xv84q\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:18.651077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:18.648503 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdb1fc5-45a9-4f6d-ab34-078c88e59025-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:19.039656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.039363 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:19.039656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.039516 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:12:19.039656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.039557 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:12:19.260658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.256736 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" event={"ID":"690fa1e2-df19-4660-a8ae-f476f18a61f7","Type":"ContainerStarted","Data":"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376"} Oct 30 22:12:19.260658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.256811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" event={"ID":"690fa1e2-df19-4660-a8ae-f476f18a61f7","Type":"ContainerStarted","Data":"8bce7e67172bbcec138574a97ddf35d114f5f3c9e9aef990df421259fb3772e9"} Oct 30 22:12:19.260658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.260448 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" Oct 30 22:12:19.263266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.263021 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8" event={"ID":"ffdb1fc5-45a9-4f6d-ab34-078c88e59025","Type":"ContainerDied","Data":"2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682"} Oct 30 22:12:19.263266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.263074 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2553192e184bcc8980613b800ccefa83113ebe7b07ed4129d68d2ab4c6632682" Oct 30 22:12:19.388599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:19.384970 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podStartSLOduration=2.384951837 podStartE2EDuration="2.384951837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.383324679 +0000 UTC m=+1067.352144246" watchObservedRunningTime="2026-10-30 22:12:19.384951837 +0000 UTC m=+1067.353771407" Oct 30 22:12:20.037555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:20.037514 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:12:20.037760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:20.037603 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:12:20.102599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:20.102199 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:21.037390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:21.037354 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:12:21.037815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:21.037406 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:12:22.037852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:22.037818 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:12:22.038230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:22.037871 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:12:23.037395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:23.037351 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:12:23.037602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:23.037418 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:12:25.016688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:25.016655 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:25.017143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:25.017123 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:25.018014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:25.017996 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:12:26.108913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.108877 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb"] Oct 30 22:12:26.109408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.109394 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ffdb1fc5-45a9-4f6d-ab34-078c88e59025" containerName="revoke-keys" Oct 30 22:12:26.109452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.109409 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdb1fc5-45a9-4f6d-ab34-078c88e59025" containerName="revoke-keys" Oct 30 22:12:26.109525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.109516 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdb1fc5-45a9-4f6d-ab34-078c88e59025" containerName="revoke-keys" Oct 30 22:12:26.112996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.112979 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.167539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.167514 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:26.167727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.167708 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:12:26.173867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.173819 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvzs\" (UniqueName: \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.173991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.173945 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.173991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.173978 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.217352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.217320 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:26.217352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.217328 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:26.220634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.220611 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb"] Oct 30 22:12:26.275026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.275001 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvzs\" (UniqueName: \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.275125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.275067 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.275125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.275091 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.275773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.275746 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.277293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.277272 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.338541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.338519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvzs\" (UniqueName: \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\") pod \"maas-api-revoke-sub-cf795471-gmmmb\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.425317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.425262 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:26.639263 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:12:26.639229 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:12:26.802907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:26.802875 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb"] Oct 30 22:12:26.803417 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:26.803393 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0e9621_1b6f_488f_8a53_ae21948ab815.slice/crio-6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8 WatchSource:0}: Error finding container 6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8: Status 404 returned error can't find the container with id 6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8 Oct 30 22:12:27.309727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.309692 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" event={"ID":"5e0e9621-1b6f-488f-8a53-ae21948ab815","Type":"ContainerStarted","Data":"29a3ee0f7dadfa7ce4ea7107b96a82a7e69155de85210338447cfcbd3b0df739"} Oct 30 22:12:27.310059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.309753 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" event={"ID":"5e0e9621-1b6f-488f-8a53-ae21948ab815","Type":"ContainerStarted","Data":"6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8"} Oct 30 22:12:27.431621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.431512 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" podStartSLOduration=1.431499504 podStartE2EDuration="1.431499504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:27.429317569 +0000 UTC m=+1075.398137149" watchObservedRunningTime="2026-10-30 22:12:27.431499504 +0000 UTC m=+1075.400319068" Oct 30 22:12:27.937343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.937304 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj"] Oct 30 22:12:27.941305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.941252 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:27.993008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.992977 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:27.993123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.993102 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:27.993193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:27.993142 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2j9v\" (UniqueName: \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.001445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.001418 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj"] Oct 30 22:12:28.094914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.094882 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.095090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.094942 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.095090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.094976 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2j9v\" (UniqueName: \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.095638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.095611 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.097287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.097269 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.102237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.102220 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:28.115210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.115187 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:28.165630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.165606 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2j9v\" (UniqueName: \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\") pod \"maas-api-revoke-sub-c2e8a213-dn5kj\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.254493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.254435 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:28.338609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.336265 4292 generic.go:356] "Generic (PLEG): container finished" podID="5e0e9621-1b6f-488f-8a53-ae21948ab815" containerID="29a3ee0f7dadfa7ce4ea7107b96a82a7e69155de85210338447cfcbd3b0df739" exitCode=0 Oct 30 22:12:28.338609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.336360 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" event={"ID":"5e0e9621-1b6f-488f-8a53-ae21948ab815","Type":"ContainerDied","Data":"29a3ee0f7dadfa7ce4ea7107b96a82a7e69155de85210338447cfcbd3b0df739"} Oct 30 22:12:28.352723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.352699 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:12:28.628846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:28.628810 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj"] Oct 30 22:12:28.629635 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:28.629560 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cacc12_d3e6_473b_af98_14e9500e9a0f.slice/crio-eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4 WatchSource:0}: Error finding container eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4: Status 404 returned error can't find the container with id eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4 Oct 30 22:12:29.343922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.343885 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" event={"ID":"15cacc12-d3e6-473b-af98-14e9500e9a0f","Type":"ContainerStarted","Data":"521be69df7d016f0e5a20ff08abbbf6b27f25210782b558b025ef458aff779d3"} Oct 30 22:12:29.344440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.343945 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" event={"ID":"15cacc12-d3e6-473b-af98-14e9500e9a0f","Type":"ContainerStarted","Data":"eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4"} Oct 30 22:12:29.465678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.465627 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" podStartSLOduration=2.465612887 podStartE2EDuration="2.465612887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:29.462956758 +0000 UTC m=+1077.431776314" watchObservedRunningTime="2026-10-30 22:12:29.465612887 +0000 UTC m=+1077.434432450" Oct 30 22:12:29.484872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.484849 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:29.511965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.511944 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\") pod \"5e0e9621-1b6f-488f-8a53-ae21948ab815\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " Oct 30 22:12:29.512074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.512009 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\") pod \"5e0e9621-1b6f-488f-8a53-ae21948ab815\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " Oct 30 22:12:29.512136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.512110 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\" (UniqueName: \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\") pod \"5e0e9621-1b6f-488f-8a53-ae21948ab815\" (UID: \"5e0e9621-1b6f-488f-8a53-ae21948ab815\") " Oct 30 22:12:29.512377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.512348 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca" pod "5e0e9621-1b6f-488f-8a53-ae21948ab815" (UID: "5e0e9621-1b6f-488f-8a53-ae21948ab815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:29.514007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.513986 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca" pod "5e0e9621-1b6f-488f-8a53-ae21948ab815" (UID: "5e0e9621-1b6f-488f-8a53-ae21948ab815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:29.514128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.514112 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs" pod "5e0e9621-1b6f-488f-8a53-ae21948ab815" (UID: "5e0e9621-1b6f-488f-8a53-ae21948ab815"). InnerVolumeSpecName "kube-api-access-hvvzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:29.613652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.613553 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:29.613652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.613627 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvvzs\" (UniqueName: \"kubernetes.io/projected/5e0e9621-1b6f-488f-8a53-ae21948ab815-kube-api-access-hvvzs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:29.613652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.613645 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e9621-1b6f-488f-8a53-ae21948ab815-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:29.765188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.765153 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9"] Oct 30 22:12:29.765686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.765672 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5e0e9621-1b6f-488f-8a53-ae21948ab815" containerName="revoke-keys" Oct 30 22:12:29.765739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.765687 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0e9621-1b6f-488f-8a53-ae21948ab815" containerName="revoke-keys" Oct 30 22:12:29.765800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.765791 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0e9621-1b6f-488f-8a53-ae21948ab815" containerName="revoke-keys" Oct 30 22:12:29.769165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.769144 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.815584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.815549 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.815688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.815649 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.815741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.815719 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlct\" (UniqueName: \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.828544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.828511 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9"] Oct 30 22:12:29.916541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.916470 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlct\" (UniqueName: \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.916716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.916622 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.916716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.916667 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.917368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.917346 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.918783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.918762 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:29.979333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:29.979305 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlct\" (UniqueName: \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\") pod \"maas-api-revoke-sub-854f9f49-cd4s9\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:30.082193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.082164 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:30.351133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.351103 4292 generic.go:356] "Generic (PLEG): container finished" podID="15cacc12-d3e6-473b-af98-14e9500e9a0f" containerID="521be69df7d016f0e5a20ff08abbbf6b27f25210782b558b025ef458aff779d3" exitCode=0 Oct 30 22:12:30.351513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.351182 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" event={"ID":"15cacc12-d3e6-473b-af98-14e9500e9a0f","Type":"ContainerDied","Data":"521be69df7d016f0e5a20ff08abbbf6b27f25210782b558b025ef458aff779d3"} Oct 30 22:12:30.352623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.352601 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" Oct 30 22:12:30.352623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.352615 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb" event={"ID":"5e0e9621-1b6f-488f-8a53-ae21948ab815","Type":"ContainerDied","Data":"6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8"} Oct 30 22:12:30.352792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.352652 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5fc358f607939f499cc1eae436b102a8b73f1f17fae4577334165716c6f6f8" Oct 30 22:12:30.459311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.459286 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9"] Oct 30 22:12:30.460058 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:30.460033 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf7e9cd_ef91_4467_8e0f_09366f3f9a92.slice/crio-232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6 WatchSource:0}: Error finding container 232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6: Status 404 returned error can't find the container with id 232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6 Oct 30 22:12:30.921420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.921380 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv"] Oct 30 22:12:30.925453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.925430 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:30.983344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:30.983286 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv"] Oct 30 22:12:31.029714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.029685 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77zl\" (UniqueName: \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.029830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.029739 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.029899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.029864 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.130837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.130804 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.130973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.130897 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77zl\" (UniqueName: \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.130973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.130925 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.131485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.131466 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.133051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.133033 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.194313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.194285 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77zl\" (UniqueName: \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\") pod \"maas-api-revoke-sub-8bc892c3-z6zlv\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.239188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.239118 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:31.362532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.362503 4292 generic.go:356] "Generic (PLEG): container finished" podID="ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" containerID="823bbed3ddc812011f571243b7a6ac1ad3e11057c0db5abec394802f9f046ee0" exitCode=0 Oct 30 22:12:31.362981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.362600 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" event={"ID":"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92","Type":"ContainerDied","Data":"823bbed3ddc812011f571243b7a6ac1ad3e11057c0db5abec394802f9f046ee0"} Oct 30 22:12:31.362981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.362642 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" event={"ID":"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92","Type":"ContainerStarted","Data":"232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6"} Oct 30 22:12:31.494113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.494048 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:31.535792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.535766 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\") pod \"15cacc12-d3e6-473b-af98-14e9500e9a0f\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " Oct 30 22:12:31.535907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.535874 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\") pod \"15cacc12-d3e6-473b-af98-14e9500e9a0f\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " Oct 30 22:12:31.535958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.535915 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\" (UniqueName: \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\") pod \"15cacc12-d3e6-473b-af98-14e9500e9a0f\" (UID: \"15cacc12-d3e6-473b-af98-14e9500e9a0f\") " Oct 30 22:12:31.536233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.536210 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca" pod "15cacc12-d3e6-473b-af98-14e9500e9a0f" (UID: "15cacc12-d3e6-473b-af98-14e9500e9a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:31.537725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.537704 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca" pod "15cacc12-d3e6-473b-af98-14e9500e9a0f" (UID: "15cacc12-d3e6-473b-af98-14e9500e9a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:31.537919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.537896 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v" pod "15cacc12-d3e6-473b-af98-14e9500e9a0f" (UID: "15cacc12-d3e6-473b-af98-14e9500e9a0f"). InnerVolumeSpecName "kube-api-access-l2j9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:31.567281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.567255 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:12:31.567793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.567779 4292 cpu_manager.go:414] "containerMap: removing container" podUID="15cacc12-d3e6-473b-af98-14e9500e9a0f" containerName="revoke-keys" Oct 30 22:12:31.567836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.567794 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cacc12-d3e6-473b-af98-14e9500e9a0f" containerName="revoke-keys" Oct 30 22:12:31.567901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.567891 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cacc12-d3e6-473b-af98-14e9500e9a0f" containerName="revoke-keys" Oct 30 22:12:31.571198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.571184 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.614206 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:31.614166 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea490d2_eb49_4d65_a661_70658bedde53.slice/crio-0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f WatchSource:0}: Error finding container 0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f: Status 404 returned error can't find the container with id 0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f Oct 30 22:12:31.614717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.614564 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv"] Oct 30 22:12:31.633295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.633265 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:12:31.637150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637125 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637156 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637175 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637312 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637380 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637442 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637483 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgzr\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637530 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.637626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637610 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:31.637626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637623 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15cacc12-d3e6-473b-af98-14e9500e9a0f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:31.637809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.637633 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2j9v\" (UniqueName: \"kubernetes.io/projected/15cacc12-d3e6-473b-af98-14e9500e9a0f-kube-api-access-l2j9v\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:31.738398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738369 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738401 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738418 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738464 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738516 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738563 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738616 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgzr\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738649 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.738751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.738671 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.739054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.739022 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.739164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.739100 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.739164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.739131 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.739253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.739203 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.739374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.739348 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.740677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.740661 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.741050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.741032 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.801852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.801829 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.801994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.801978 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgzr\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:31.886321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:31.886293 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:32.283857 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:32.283828 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6618093_a44d_42a7_b805_2a9282c9d057.slice/crio-64aa3095588279eb6dd5e3b45ae33ff81e9365b2d284feb7eb74d0e30a6c5c03 WatchSource:0}: Error finding container 64aa3095588279eb6dd5e3b45ae33ff81e9365b2d284feb7eb74d0e30a6c5c03: Status 404 returned error can't find the container with id 64aa3095588279eb6dd5e3b45ae33ff81e9365b2d284feb7eb74d0e30a6c5c03 Oct 30 22:12:32.284934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.284905 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:12:32.286194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.286160 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:32.286293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.286235 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:32.286293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.286279 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:32.372109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.372074 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" event={"ID":"e6618093-a44d-42a7-b805-2a9282c9d057","Type":"ContainerStarted","Data":"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a"} Oct 30 22:12:32.372483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.372141 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" event={"ID":"e6618093-a44d-42a7-b805-2a9282c9d057","Type":"ContainerStarted","Data":"64aa3095588279eb6dd5e3b45ae33ff81e9365b2d284feb7eb74d0e30a6c5c03"} Oct 30 22:12:32.374027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.373995 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" event={"ID":"dea490d2-eb49-4d65-a661-70658bedde53","Type":"ContainerStarted","Data":"487bcd3f92a4378c41faacbae4e44920422cfafed230fb837807f3d990e5dd9e"} Oct 30 22:12:32.374167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.374039 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" event={"ID":"dea490d2-eb49-4d65-a661-70658bedde53","Type":"ContainerStarted","Data":"0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f"} Oct 30 22:12:32.375620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.375570 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" Oct 30 22:12:32.375620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.375610 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj" event={"ID":"15cacc12-d3e6-473b-af98-14e9500e9a0f","Type":"ContainerDied","Data":"eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4"} Oct 30 22:12:32.375925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.375650 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb723d909a1b1c5c93037678bd17d2b21b5ce5b8edfbba17534065e2504609d4" Oct 30 22:12:32.526855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.526805 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podStartSLOduration=1.526791576 podStartE2EDuration="1.526791576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.508013634 +0000 UTC m=+1080.476833227" watchObservedRunningTime="2026-10-30 22:12:32.526791576 +0000 UTC m=+1080.495611140" Oct 30 22:12:32.532910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.532883 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:32.650172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.650143 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\") pod \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " Oct 30 22:12:32.650309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.650268 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\" (UniqueName: \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\") pod \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " Oct 30 22:12:32.650469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.650451 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\") pod \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\" (UID: \"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92\") " Oct 30 22:12:32.650549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.650512 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca" pod "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" (UID: "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:32.650826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.650809 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:32.652310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.652288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca" pod "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" (UID: "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:32.652391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.652331 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct" pod "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" (UID: "ecf7e9cd-ef91-4467-8e0f-09366f3f9a92"). InnerVolumeSpecName "kube-api-access-8jlct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:32.694134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.694036 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" podStartSLOduration=2.694023139 podStartE2EDuration="2.694023139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.691784016 +0000 UTC m=+1080.660603585" watchObservedRunningTime="2026-10-30 22:12:32.694023139 +0000 UTC m=+1080.662842759" Oct 30 22:12:32.751982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.751947 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:32.751982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.751975 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jlct\" (UniqueName: \"kubernetes.io/projected/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92-kube-api-access-8jlct\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:32.817651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.817610 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s"] Oct 30 22:12:32.818209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.818191 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" containerName="revoke-keys" Oct 30 22:12:32.818262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.818214 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" containerName="revoke-keys" Oct 30 22:12:32.818362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.818350 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" containerName="revoke-keys" Oct 30 22:12:32.822390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.822364 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:32.881609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.881548 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s"] Oct 30 22:12:32.908884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.908827 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:32.954009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.953920 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:32.954009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.953980 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754cl\" (UniqueName: \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:32.954181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:32.954065 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.055407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.055367 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754cl\" (UniqueName: \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.055407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.055413 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.055695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.055553 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.056185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.056158 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.057773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.057755 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.119727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.119695 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-754cl\" (UniqueName: \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\") pod \"maas-api-revoke-sub-d30b8795-tsw6s\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.136283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.136260 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:33.418100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.418066 4292 generic.go:356] "Generic (PLEG): container finished" podID="dea490d2-eb49-4d65-a661-70658bedde53" containerID="487bcd3f92a4378c41faacbae4e44920422cfafed230fb837807f3d990e5dd9e" exitCode=0 Oct 30 22:12:33.418554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.418151 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" event={"ID":"dea490d2-eb49-4d65-a661-70658bedde53","Type":"ContainerDied","Data":"487bcd3f92a4378c41faacbae4e44920422cfafed230fb837807f3d990e5dd9e"} Oct 30 22:12:33.419746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.419719 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" event={"ID":"ecf7e9cd-ef91-4467-8e0f-09366f3f9a92","Type":"ContainerDied","Data":"232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6"} Oct 30 22:12:33.419843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.419770 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232403a7e7315df87f44822872507463365070b4c6664fb3dc8954e1f2a1b6e6" Oct 30 22:12:33.419888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.419876 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9" Oct 30 22:12:33.534324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.534295 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s"] Oct 30 22:12:33.534839 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:33.534812 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ee8525_eafb_462b_b69b_78937278eae4.slice/crio-ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0 WatchSource:0}: Error finding container ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0: Status 404 returned error can't find the container with id ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0 Oct 30 22:12:33.907605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.901694 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:33.907605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:33.901752 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:34.428074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.428036 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" event={"ID":"d1ee8525-eafb-462b-b69b-78937278eae4","Type":"ContainerStarted","Data":"dc5ffeed3f2d616aa43a3f3749d7bdba7abbd6af307c8dc6fb24795ff1f1ffa6"} Oct 30 22:12:34.428471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.428100 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" event={"ID":"d1ee8525-eafb-462b-b69b-78937278eae4","Type":"ContainerStarted","Data":"ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0"} Oct 30 22:12:34.559201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.559145 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" podStartSLOduration=2.559131886 podStartE2EDuration="2.559131886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.557927513 +0000 UTC m=+1082.526747083" watchObservedRunningTime="2026-10-30 22:12:34.559131886 +0000 UTC m=+1082.527951525" Oct 30 22:12:34.617621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.617593 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:34.675941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.675912 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\" (UniqueName: \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\") pod \"dea490d2-eb49-4d65-a661-70658bedde53\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " Oct 30 22:12:34.676084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.676068 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\") pod \"dea490d2-eb49-4d65-a661-70658bedde53\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " Oct 30 22:12:34.676226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.676211 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\") pod \"dea490d2-eb49-4d65-a661-70658bedde53\" (UID: \"dea490d2-eb49-4d65-a661-70658bedde53\") " Oct 30 22:12:34.676596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.676552 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca" pod "dea490d2-eb49-4d65-a661-70658bedde53" (UID: "dea490d2-eb49-4d65-a661-70658bedde53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:34.677983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.677955 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl" pod "dea490d2-eb49-4d65-a661-70658bedde53" (UID: "dea490d2-eb49-4d65-a661-70658bedde53"). InnerVolumeSpecName "kube-api-access-g77zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:34.678070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.678031 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca" pod "dea490d2-eb49-4d65-a661-70658bedde53" (UID: "dea490d2-eb49-4d65-a661-70658bedde53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:34.777361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.777331 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:34.777361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.777359 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g77zl\" (UniqueName: \"kubernetes.io/projected/dea490d2-eb49-4d65-a661-70658bedde53-kube-api-access-g77zl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:34.777531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.777370 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea490d2-eb49-4d65-a661-70658bedde53-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:34.895131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.893974 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:34.895131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:34.894031 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:35.436413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.436350 4292 generic.go:356] "Generic (PLEG): container finished" podID="d1ee8525-eafb-462b-b69b-78937278eae4" containerID="dc5ffeed3f2d616aa43a3f3749d7bdba7abbd6af307c8dc6fb24795ff1f1ffa6" exitCode=0 Oct 30 22:12:35.436413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.436403 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" event={"ID":"d1ee8525-eafb-462b-b69b-78937278eae4","Type":"ContainerDied","Data":"dc5ffeed3f2d616aa43a3f3749d7bdba7abbd6af307c8dc6fb24795ff1f1ffa6"} Oct 30 22:12:35.438287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.438260 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" Oct 30 22:12:35.438388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.438292 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv" event={"ID":"dea490d2-eb49-4d65-a661-70658bedde53","Type":"ContainerDied","Data":"0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f"} Oct 30 22:12:35.438388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.438341 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0514638064c1cb061ff33d357ce34e294106ca3499a4c490b1ab4966f2cb859f" Oct 30 22:12:35.893599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.888970 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:35.893599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:35.889033 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:36.640125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.640099 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:36.726902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.726819 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\" (UniqueName: \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\") pod \"d1ee8525-eafb-462b-b69b-78937278eae4\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " Oct 30 22:12:36.727069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.726919 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\") pod \"d1ee8525-eafb-462b-b69b-78937278eae4\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " Oct 30 22:12:36.727069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.727006 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\") pod \"d1ee8525-eafb-462b-b69b-78937278eae4\" (UID: \"d1ee8525-eafb-462b-b69b-78937278eae4\") " Oct 30 22:12:36.727367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.727340 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca" pod "d1ee8525-eafb-462b-b69b-78937278eae4" (UID: "d1ee8525-eafb-462b-b69b-78937278eae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:36.727483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.727424 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.729009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.728983 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca" pod "d1ee8525-eafb-462b-b69b-78937278eae4" (UID: "d1ee8525-eafb-462b-b69b-78937278eae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:36.729127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.729090 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl" pod "d1ee8525-eafb-462b-b69b-78937278eae4" (UID: "d1ee8525-eafb-462b-b69b-78937278eae4"). InnerVolumeSpecName "kube-api-access-754cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:36.829169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.829122 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-754cl\" (UniqueName: \"kubernetes.io/projected/d1ee8525-eafb-462b-b69b-78937278eae4-kube-api-access-754cl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.829169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.829170 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ee8525-eafb-462b-b69b-78937278eae4-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.909600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.907043 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:36.909600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:36.907107 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:37.451631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:37.451600 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" Oct 30 22:12:37.451806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:37.451632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s" event={"ID":"d1ee8525-eafb-462b-b69b-78937278eae4","Type":"ContainerDied","Data":"ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0"} Oct 30 22:12:37.451806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:37.451680 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad97c4b0bffe4621df052ab6c3df8435ad2a66075e6bb74b86abc77bcfd355a0" Oct 30 22:12:37.902597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:37.894897 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:37.902597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:37.894964 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:38.894414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:38.893959 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:12:38.894414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:38.894026 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:12:40.014671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.014637 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2"] Oct 30 22:12:40.020202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020175 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d1ee8525-eafb-462b-b69b-78937278eae4" containerName="revoke-keys" Oct 30 22:12:40.020357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020341 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ee8525-eafb-462b-b69b-78937278eae4" containerName="revoke-keys" Oct 30 22:12:40.020507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020493 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dea490d2-eb49-4d65-a661-70658bedde53" containerName="revoke-keys" Oct 30 22:12:40.020625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020611 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea490d2-eb49-4d65-a661-70658bedde53" containerName="revoke-keys" Oct 30 22:12:40.020876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020860 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ee8525-eafb-462b-b69b-78937278eae4" containerName="revoke-keys" Oct 30 22:12:40.020996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.020982 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea490d2-eb49-4d65-a661-70658bedde53" containerName="revoke-keys" Oct 30 22:12:40.028035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.028017 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.075536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.075504 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2"] Oct 30 22:12:40.149628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.149565 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:40.150079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.150032 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:12:40.150652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.150618 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.150785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.150766 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wq7n\" (UniqueName: \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.150937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.150922 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.154440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.154416 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:40.172602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.172550 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:40.187265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.187239 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:40.188098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.188076 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:40.188737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.188705 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:12:40.252369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.252334 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.252561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.252375 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wq7n\" (UniqueName: \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.252561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.252423 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.253113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.253086 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.255298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.255274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.318649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.318610 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wq7n\" (UniqueName: \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\") pod \"maas-api-revoke-sub-74908f82-rz9m2\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.366347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.366310 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:40.753030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:40.753005 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2"] Oct 30 22:12:40.755063 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:40.755038 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f5c000_894f_441f_a79d_2aed7a8bd8f6.slice/crio-21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea WatchSource:0}: Error finding container 21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea: Status 404 returned error can't find the container with id 21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea Oct 30 22:12:41.517973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:41.517932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" event={"ID":"82f5c000-894f-441f-a79d-2aed7a8bd8f6","Type":"ContainerStarted","Data":"686e7d18d33f54b8c692598700e755715cf9df9743e01682627959375e954510"} Oct 30 22:12:41.518453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:41.517989 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" event={"ID":"82f5c000-894f-441f-a79d-2aed7a8bd8f6","Type":"ContainerStarted","Data":"21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea"} Oct 30 22:12:41.638438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:41.638392 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" podStartSLOduration=2.638379503 podStartE2EDuration="2.638379503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:41.638081949 +0000 UTC m=+1089.606901514" watchObservedRunningTime="2026-10-30 22:12:41.638379503 +0000 UTC m=+1089.607199067" Oct 30 22:12:42.525881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:42.525838 4292 generic.go:356] "Generic (PLEG): container finished" podID="82f5c000-894f-441f-a79d-2aed7a8bd8f6" containerID="686e7d18d33f54b8c692598700e755715cf9df9743e01682627959375e954510" exitCode=0 Oct 30 22:12:42.526409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:42.525885 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" event={"ID":"82f5c000-894f-441f-a79d-2aed7a8bd8f6","Type":"ContainerDied","Data":"686e7d18d33f54b8c692598700e755715cf9df9743e01682627959375e954510"} Oct 30 22:12:43.681763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.681739 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:43.792045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.791974 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\" (UniqueName: \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\") pod \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " Oct 30 22:12:43.792045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.792013 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\") pod \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " Oct 30 22:12:43.792045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.792033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\") pod \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\" (UID: \"82f5c000-894f-441f-a79d-2aed7a8bd8f6\") " Oct 30 22:12:43.792455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.792428 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca" pod "82f5c000-894f-441f-a79d-2aed7a8bd8f6" (UID: "82f5c000-894f-441f-a79d-2aed7a8bd8f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:43.794137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.794105 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca" pod "82f5c000-894f-441f-a79d-2aed7a8bd8f6" (UID: "82f5c000-894f-441f-a79d-2aed7a8bd8f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:43.794257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.794221 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n" pod "82f5c000-894f-441f-a79d-2aed7a8bd8f6" (UID: "82f5c000-894f-441f-a79d-2aed7a8bd8f6"). InnerVolumeSpecName "kube-api-access-7wq7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:43.893807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.893777 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wq7n\" (UniqueName: \"kubernetes.io/projected/82f5c000-894f-441f-a79d-2aed7a8bd8f6-kube-api-access-7wq7n\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:43.893807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.893802 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:43.893807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:43.893812 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f5c000-894f-441f-a79d-2aed7a8bd8f6-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:44.538962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:44.538885 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" Oct 30 22:12:44.538962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:44.538922 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2" event={"ID":"82f5c000-894f-441f-a79d-2aed7a8bd8f6","Type":"ContainerDied","Data":"21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea"} Oct 30 22:12:44.539151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:44.538998 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b2b6a4139dc0ecaeeedcbeff38cc0421f0257de8d86b81105219ced5c7b9ea" Oct 30 22:12:48.480146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.480109 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg"] Oct 30 22:12:48.480847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.480826 4292 cpu_manager.go:414] "containerMap: removing container" podUID="82f5c000-894f-441f-a79d-2aed7a8bd8f6" containerName="revoke-keys" Oct 30 22:12:48.480960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.480848 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f5c000-894f-441f-a79d-2aed7a8bd8f6" containerName="revoke-keys" Oct 30 22:12:48.481043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.481029 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f5c000-894f-441f-a79d-2aed7a8bd8f6" containerName="revoke-keys" Oct 30 22:12:48.485653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.485624 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.543158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.543126 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.543321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.543286 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9c2\" (UniqueName: \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.543321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.543318 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.544986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.544959 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg"] Oct 30 22:12:48.590741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.590718 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:48.591033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.591016 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:48.591090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.591036 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:48.591125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.591111 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:12:48.643991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.643968 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9c2\" (UniqueName: \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.644115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.644003 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.644115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.644106 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.644725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.644703 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.646372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.646351 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.706810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.706790 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9c2\" (UniqueName: \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\") pod \"maas-api-revoke-sub-1f7d6bf6-kxktg\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:48.799034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:48.799012 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:49.058093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.058013 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:12:49.058265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.058230 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" containerID="cri-o://ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57" gracePeriod=30 Oct 30 22:12:49.061480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.061453 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:12:49.061662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.061628 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" containerID="cri-o://66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7" gracePeriod=30 Oct 30 22:12:49.178007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.177972 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg"] Oct 30 22:12:49.178168 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:49.178139 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8179c7f_678d_464a_9c0b_8143c3f33d35.slice/crio-095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5 WatchSource:0}: Error finding container 095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5: Status 404 returned error can't find the container with id 095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5 Oct 30 22:12:49.571695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.571663 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" event={"ID":"a8179c7f-678d-464a-9c0b-8143c3f33d35","Type":"ContainerStarted","Data":"8a34e4e0b69dcfbb39b51b7dee49e9fd2e89c1e048bea7520a6f263ed34c42ae"} Oct 30 22:12:49.572046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.571719 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" event={"ID":"a8179c7f-678d-464a-9c0b-8143c3f33d35","Type":"ContainerStarted","Data":"095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5"} Oct 30 22:12:49.695209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:49.695129 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" podStartSLOduration=1.695115787 podStartE2EDuration="1.695115787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:49.691354327 +0000 UTC m=+1097.660173891" watchObservedRunningTime="2026-10-30 22:12:49.695115787 +0000 UTC m=+1097.663935350" Oct 30 22:12:50.580128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:50.580096 4292 generic.go:356] "Generic (PLEG): container finished" podID="a8179c7f-678d-464a-9c0b-8143c3f33d35" containerID="8a34e4e0b69dcfbb39b51b7dee49e9fd2e89c1e048bea7520a6f263ed34c42ae" exitCode=0 Oct 30 22:12:50.580503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:50.580188 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" event={"ID":"a8179c7f-678d-464a-9c0b-8143c3f33d35","Type":"ContainerDied","Data":"8a34e4e0b69dcfbb39b51b7dee49e9fd2e89c1e048bea7520a6f263ed34c42ae"} Oct 30 22:12:51.751614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.751589 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:51.876891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.876812 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\") pod \"a8179c7f-678d-464a-9c0b-8143c3f33d35\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " Oct 30 22:12:51.877027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.876913 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\" (UniqueName: \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\") pod \"a8179c7f-678d-464a-9c0b-8143c3f33d35\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " Oct 30 22:12:51.877027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.876944 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\") pod \"a8179c7f-678d-464a-9c0b-8143c3f33d35\" (UID: \"a8179c7f-678d-464a-9c0b-8143c3f33d35\") " Oct 30 22:12:51.877372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.877338 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca" pod "a8179c7f-678d-464a-9c0b-8143c3f33d35" (UID: "a8179c7f-678d-464a-9c0b-8143c3f33d35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:51.878882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.878853 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca" pod "a8179c7f-678d-464a-9c0b-8143c3f33d35" (UID: "a8179c7f-678d-464a-9c0b-8143c3f33d35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:51.878996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.878979 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2" pod "a8179c7f-678d-464a-9c0b-8143c3f33d35" (UID: "a8179c7f-678d-464a-9c0b-8143c3f33d35"). InnerVolumeSpecName "kube-api-access-rs9c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:51.978813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.978778 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:51.978813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.978810 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9c2\" (UniqueName: \"kubernetes.io/projected/a8179c7f-678d-464a-9c0b-8143c3f33d35-kube-api-access-rs9c2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:51.979036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:51.978824 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8179c7f-678d-464a-9c0b-8143c3f33d35-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.593074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:52.593047 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" Oct 30 22:12:52.593239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:52.593049 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg" event={"ID":"a8179c7f-678d-464a-9c0b-8143c3f33d35","Type":"ContainerDied","Data":"095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5"} Oct 30 22:12:52.593239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:52.593170 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="095e900a65f0cc40c80042ecf7b6a955a1c995bc6e1ebd3352d22fed17d0c9f5" Oct 30 22:12:54.320457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.320378 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:12:54.380659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.380634 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:12:54.403953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.403929 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404005 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404048 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404087 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404137 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404196 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404243 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404273 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404299 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\") pod \"0ff748f4-e860-4c3a-98f5-0e199853b238\" (UID: \"0ff748f4-e860-4c3a-98f5-0e199853b238\") " Oct 30 22:12:54.404654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404296 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.404654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404418 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.404654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404432 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:54.404654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404502 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.404654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404525 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.404970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404906 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.404970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404930 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.404970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404945 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff748f4-e860-4c3a-98f5-0e199853b238-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.404970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404959 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.405184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.404973 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.406412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.406383 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.406412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.406399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:54.406532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.406433 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:54.406532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.406490 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb" pod "0ff748f4-e860-4c3a-98f5-0e199853b238" (UID: "0ff748f4-e860-4c3a-98f5-0e199853b238"). InnerVolumeSpecName "kube-api-access-n6ccb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:54.506021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.505992 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506129 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506172 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506209 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506249 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506282 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506341 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506372 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506436 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\") pod \"81598a95-0804-4754-b8b3-c3328274f42f\" (UID: \"81598a95-0804-4754-b8b3-c3328274f42f\") " Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506448 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506478 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506690 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.506826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506761 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.506978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507122 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507146 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ccb\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-kube-api-access-n6ccb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507163 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507175 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507188 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507202 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507215 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507228 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff748f4-e860-4c3a-98f5-0e199853b238-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.507389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.507241 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81598a95-0804-4754-b8b3-c3328274f42f-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.508734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.508706 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:54.508886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.508870 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:54.509123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.509107 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "kube-api-access-vxsld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:54.509183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.509171 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token" pod "81598a95-0804-4754-b8b3-c3328274f42f" (UID: "81598a95-0804-4754-b8b3-c3328274f42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:54.608278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608199 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.608278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608230 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81598a95-0804-4754-b8b3-c3328274f42f-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.608278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608242 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81598a95-0804-4754-b8b3-c3328274f42f-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.608278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608255 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxsld\" (UniqueName: \"kubernetes.io/projected/81598a95-0804-4754-b8b3-c3328274f42f-kube-api-access-vxsld\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:54.608629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608287 4292 generic.go:356] "Generic (PLEG): container finished" podID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerID="ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57" exitCode=0 Oct 30 22:12:54.608629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608334 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" Oct 30 22:12:54.608629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608370 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" event={"ID":"0ff748f4-e860-4c3a-98f5-0e199853b238","Type":"ContainerDied","Data":"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57"} Oct 30 22:12:54.608629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608418 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj" event={"ID":"0ff748f4-e860-4c3a-98f5-0e199853b238","Type":"ContainerDied","Data":"9f52619b37863af00e24631d90036733e970364fae309c96a98b00bffaf56fd1"} Oct 30 22:12:54.608629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.608447 4292 scope.go:122] "RemoveContainer" containerID="ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57" Oct 30 22:12:54.610206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.610186 4292 generic.go:356] "Generic (PLEG): container finished" podID="81598a95-0804-4754-b8b3-c3328274f42f" containerID="66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7" exitCode=0 Oct 30 22:12:54.610340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.610239 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" Oct 30 22:12:54.610340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.610287 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" event={"ID":"81598a95-0804-4754-b8b3-c3328274f42f","Type":"ContainerDied","Data":"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7"} Oct 30 22:12:54.610426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.610331 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v" event={"ID":"81598a95-0804-4754-b8b3-c3328274f42f","Type":"ContainerDied","Data":"4b6d77e711f89e4957fb698118bdc0ab2798a55241ba0ac52ead3cb3d7c2ddec"} Oct 30 22:12:54.621747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.621730 4292 scope.go:122] "RemoveContainer" containerID="ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57" Oct 30 22:12:54.622039 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:12:54.622018 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57\": container with ID starting with ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57 not found: ID does not exist" containerID="ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57" Oct 30 22:12:54.622106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.622048 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57"} err="failed to get container status \"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57\": rpc error: code = NotFound desc = could not find container \"ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57\": container with ID starting with ef670e6cd680e847c044cfcf743ad23e4725e94801131bcce4da16c4c522bb57 not found: ID does not exist" Oct 30 22:12:54.622106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.622064 4292 scope.go:122] "RemoveContainer" containerID="66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7" Oct 30 22:12:54.631879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.631862 4292 scope.go:122] "RemoveContainer" containerID="66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7" Oct 30 22:12:54.632121 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:12:54.632102 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7\": container with ID starting with 66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7 not found: ID does not exist" containerID="66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7" Oct 30 22:12:54.632194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.632129 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7"} err="failed to get container status \"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7\": rpc error: code = NotFound desc = could not find container \"66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7\": container with ID starting with 66a19bb066643dbcdc3cc50caffc3595c5d708b9e09419a01b291bedb0b4ddb7 not found: ID does not exist" Oct 30 22:12:54.788945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.788910 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:12:54.795819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.795791 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-pw4rj"] Oct 30 22:12:54.909285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.909212 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:12:54.915909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:54.915882 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-stk8v"] Oct 30 22:12:56.146144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:56.146100 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" path="/var/lib/kubelet/pods/0ff748f4-e860-4c3a-98f5-0e199853b238/volumes" Oct 30 22:12:56.146647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:56.146529 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81598a95-0804-4754-b8b3-c3328274f42f" path="/var/lib/kubelet/pods/81598a95-0804-4754-b8b3-c3328274f42f/volumes" Oct 30 22:12:56.640965 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:12:56.640929 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:12:58.362607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.362561 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:12:58.363134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363118 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" Oct 30 22:12:58.363134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363133 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" Oct 30 22:12:58.363234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363145 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a8179c7f-678d-464a-9c0b-8143c3f33d35" containerName="revoke-keys" Oct 30 22:12:58.363234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363151 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8179c7f-678d-464a-9c0b-8143c3f33d35" containerName="revoke-keys" Oct 30 22:12:58.363234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363161 4292 cpu_manager.go:414] "containerMap: removing container" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" Oct 30 22:12:58.363234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363166 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" Oct 30 22:12:58.363360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363257 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff748f4-e860-4c3a-98f5-0e199853b238" containerName="istio-proxy" Oct 30 22:12:58.363360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363264 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8179c7f-678d-464a-9c0b-8143c3f33d35" containerName="revoke-keys" Oct 30 22:12:58.363360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.363272 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="81598a95-0804-4754-b8b3-c3328274f42f" containerName="istio-proxy" Oct 30 22:12:58.368433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.368411 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.423469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.423426 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-77cb1f-openshift-default-dockercfg-ttk97\"" Oct 30 22:12:58.471781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.471744 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5r8\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.471981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.471811 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.471981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.471933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.471981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.471975 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.472167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.472054 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.472167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.472124 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.472270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.472236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.472345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.472327 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.472405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.472387 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.477148 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.477123 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:12:58.573781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.573751 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.573974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.573820 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.573974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.573870 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.573974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.573914 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.573995 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5r8\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574058 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574098 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574128 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574148 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574239 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574294 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574528 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.574817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.574793 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.576254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.576232 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.576520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.576500 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.641078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.640780 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.641660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.641603 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5r8\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:58.690816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:58.690792 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:12:59.128819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.128788 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:12:59.129633 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:12:59.129557 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c275a4_0c6f_469a_815d_d0cb228b5cc0.slice/crio-e543ca179cb38e4f04426a58fdd89dfb78e5a7b411a1f6d2662e747613e7cb0c WatchSource:0}: Error finding container e543ca179cb38e4f04426a58fdd89dfb78e5a7b411a1f6d2662e747613e7cb0c: Status 404 returned error can't find the container with id e543ca179cb38e4f04426a58fdd89dfb78e5a7b411a1f6d2662e747613e7cb0c Oct 30 22:12:59.132307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.132276 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:59.132387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.132342 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:59.132387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.132371 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:12:59.666075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.666042 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" event={"ID":"42c275a4-0c6f-469a-815d-d0cb228b5cc0","Type":"ContainerStarted","Data":"e9e8143430e66ad61c90e63ca17887d0ce1087691cb8c14f2cb6ee10af72c812"} Oct 30 22:12:59.666512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.666098 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" event={"ID":"42c275a4-0c6f-469a-815d-d0cb228b5cc0","Type":"ContainerStarted","Data":"e543ca179cb38e4f04426a58fdd89dfb78e5a7b411a1f6d2662e747613e7cb0c"} Oct 30 22:12:59.692130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:12:59.692100 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:13:00.703726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:00.703690 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:00.704296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:00.704262 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:01.705597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:01.705152 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:01.705597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:01.705208 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:02.693301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:02.693267 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:02.693591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:02.693541 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:03.708597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:03.708014 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:03.708597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:03.708086 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:04.694596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:04.693859 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:04.694596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:04.693930 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:05.691251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:05.691211 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:13:05.691647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:05.691275 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:13:06.695643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:06.695609 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:13:06.696082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:06.695968 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:13:06.696710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:06.696691 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:13:06.875364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:06.875318 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podStartSLOduration=8.875305938 podStartE2EDuration="8.875305938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:59.811734575 +0000 UTC m=+1107.780554140" watchObservedRunningTime="2026-10-30 22:13:06.875305938 +0000 UTC m=+1114.844125502" Oct 30 22:13:13.360527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.360490 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:13:13.365815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.365793 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.425699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.425665 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:13:13.450504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450479 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.450656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450516 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.450656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450547 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.450775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450720 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzjn\" (UniqueName: \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.450867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450850 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.450927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.450887 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.471350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.471327 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2a4123\"/\"kube-root-ca.crt\"" Oct 30 22:13:13.471350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.471338 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2a4123\"/\"openshift-service-ca.crt\"" Oct 30 22:13:13.471521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.471328 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2a4123\"/\"distinct-w0-2a4123-kserve-self-signed-certs\"" Oct 30 22:13:13.471596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.471557 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2a4123\"/\"default-dockercfg-snr45\"" Oct 30 22:13:13.551726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551689 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.551726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551734 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551800 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551822 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551856 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.551892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzjn\" (UniqueName: \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.552146 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.552202 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.552321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.552258 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.554098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.554079 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.554271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.554253 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.615351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.615286 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzjn\" (UniqueName: \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\") pod \"distinct-w0-2a4123-kserve-9cd579688-8dr99\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:13.678397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:13.678370 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:14.059724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:14.059692 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:13:14.059872 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:13:14.059738 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7508f48e_dc12_4184_8fe8_a243ad9fd4aa.slice/crio-7be635b66508e681489861fcabf157ce1a17bcfc66cdc8e88a833f2ee906b928 WatchSource:0}: Error finding container 7be635b66508e681489861fcabf157ce1a17bcfc66cdc8e88a833f2ee906b928: Status 404 returned error can't find the container with id 7be635b66508e681489861fcabf157ce1a17bcfc66cdc8e88a833f2ee906b928 Oct 30 22:13:14.802377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:14.802337 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" event={"ID":"7508f48e-dc12-4184-8fe8-a243ad9fd4aa","Type":"ContainerStarted","Data":"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049"} Oct 30 22:13:14.802778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:14.802389 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" event={"ID":"7508f48e-dc12-4184-8fe8-a243ad9fd4aa","Type":"ContainerStarted","Data":"7be635b66508e681489861fcabf157ce1a17bcfc66cdc8e88a833f2ee906b928"} Oct 30 22:13:14.925072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:14.925019 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podStartSLOduration=1.9250047970000002 podStartE2EDuration="1.925004797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:14.922908859 +0000 UTC m=+1122.891728425" watchObservedRunningTime="2026-10-30 22:13:14.925004797 +0000 UTC m=+1122.893824409" Oct 30 22:13:15.678970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:15.678930 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:23.678801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:23.678766 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:23.692334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:23.692311 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:23.872355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:23.872325 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:13:26.639725 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:13:26.639688 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:13:38.660150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.660119 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:13:38.665475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.665456 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.726857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.726826 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:13:38.770595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.770541 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-13b8d8\"/\"openshift-service-ca.crt\"" Oct 30 22:13:38.770749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.770671 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-13b8d8\"/\"kube-root-ca.crt\"" Oct 30 22:13:38.770749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.770701 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-13b8d8\"/\"distinct-w2-13b8d8-kserve-self-signed-certs\"" Oct 30 22:13:38.770838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.770752 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-13b8d8\"/\"default-dockercfg-9vlnw\"" Oct 30 22:13:38.814871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.814832 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.815059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.814889 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.815059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.814930 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.815059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.815017 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.815059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.815043 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lssp\" (UniqueName: \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.815227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.815068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.915986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.915901 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.915986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.915937 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.915986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.915981 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916011 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lssp\" (UniqueName: \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916048 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916248 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916344 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916326 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.916465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.916438 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.918348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.918322 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.918504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.918483 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.979346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.979320 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lssp\" (UniqueName: \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\") pod \"distinct-w2-13b8d8-kserve-88886576f-nwhc4\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:38.979889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:38.979869 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:39.384665 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:13:39.384633 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb09c3d0_e9ac_4a5b_b80b_3d693df346b5.slice/crio-ca7e345abf114d7d11119675371a2b546da709cce18eca52fdb217e43495a3f3 WatchSource:0}: Error finding container ca7e345abf114d7d11119675371a2b546da709cce18eca52fdb217e43495a3f3: Status 404 returned error can't find the container with id ca7e345abf114d7d11119675371a2b546da709cce18eca52fdb217e43495a3f3 Oct 30 22:13:39.384810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:39.384792 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:13:39.962589 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:39.962545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" event={"ID":"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5","Type":"ContainerStarted","Data":"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2"} Oct 30 22:13:39.962970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:39.962600 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" event={"ID":"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5","Type":"ContainerStarted","Data":"ca7e345abf114d7d11119675371a2b546da709cce18eca52fdb217e43495a3f3"} Oct 30 22:13:39.980322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:39.980297 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:40.083949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:40.083901 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podStartSLOduration=2.083887539 podStartE2EDuration="2.083887539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:40.082561252 +0000 UTC m=+1148.051380817" watchObservedRunningTime="2026-10-30 22:13:40.083887539 +0000 UTC m=+1148.052707103" Oct 30 22:13:47.652518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.652482 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:13:47.656699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.656681 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.710874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.710845 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.711060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.710914 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.711140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.711061 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6jx\" (UniqueName: \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.711140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.711108 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.711489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.711356 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.711631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.711612 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.762566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.762537 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c76d9e\"/\"distinct-w6-c76d9e-kserve-self-signed-certs\"" Oct 30 22:13:47.766244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.766221 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:13:47.813075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.813040 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.813392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.813368 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.813613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.813594 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.813849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.813831 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.815371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.815338 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.815616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.815595 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6jx\" (UniqueName: \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.815809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.814859 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.816016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.814556 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.816171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.814339 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.819930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.819903 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.822163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.822131 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.879452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.879420 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6jx\" (UniqueName: \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\") pod \"distinct-w6-c76d9e-kserve-54699674bf-7nkpg\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:47.969851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:47.969785 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:48.352191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:48.352153 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:13:48.980466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:48.980438 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:48.993306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:48.993275 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:49.022829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:49.022797 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" event={"ID":"132667f4-0f29-44a0-8951-4e7ac378320c","Type":"ContainerStarted","Data":"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd"} Oct 30 22:13:49.022962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:49.022840 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" event={"ID":"132667f4-0f29-44a0-8951-4e7ac378320c","Type":"ContainerStarted","Data":"d83c0716fc61c0aa6b2c113e5161133071adf8c3387d5fc3c6903aede7adc435"} Oct 30 22:13:49.033954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:49.033933 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:13:49.240280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:49.240186 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podStartSLOduration=2.240173613 podStartE2EDuration="2.240173613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.236971471 +0000 UTC m=+1157.205791035" watchObservedRunningTime="2026-10-30 22:13:49.240173613 +0000 UTC m=+1157.208993177" Oct 30 22:13:49.970542 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:49.970512 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:56.641293 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:13:56.641249 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:13:57.970856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:57.970814 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:57.984731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:57.984705 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:13:58.101170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:13:58.101140 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:14:17.487291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.487254 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:14:17.491399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.491380 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540501 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540553 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540673 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540720 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540787 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sf6\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540837 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540861 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.540988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.540896 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642294 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sf6\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642370 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642554 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642625 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642683 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642737 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642773 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.642970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642861 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.643072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.642982 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.643204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.643180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.643306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.643283 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.643531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.643507 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.645091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.645065 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.645203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.645157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.706675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.706644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sf6\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.707745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.707701 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:14:17.709057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.709032 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:17.806956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:17.806925 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:18.147936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.147866 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:14:18.152769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.152749 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.213607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.211360 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:14:18.249368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249342 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pb7\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249389 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249748 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249835 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249864 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249886 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.249982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.249938 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.250038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.250027 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.291379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.291356 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:14:18.292234 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:14:18.292210 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d90424d_40d1_4bfd_9780_b3b36b0be4ff.slice/crio-01fe347a42b0d6ec21e52aa39eb4e1ffbf19bc6e1e00d2557c6b9bc4e3504676 WatchSource:0}: Error finding container 01fe347a42b0d6ec21e52aa39eb4e1ffbf19bc6e1e00d2557c6b9bc4e3504676: Status 404 returned error can't find the container with id 01fe347a42b0d6ec21e52aa39eb4e1ffbf19bc6e1e00d2557c6b9bc4e3504676 Oct 30 22:14:18.294698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.294669 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:18.294764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.294727 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:18.294764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.294757 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:18.351401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351367 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351483 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pb7\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351539 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351642 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351731 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351751 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351779 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351812 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351859 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.351919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351873 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.352067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.351919 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.352192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.352146 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.352277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.352242 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.352789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.352764 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.353869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.353850 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.354178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.354157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.428662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.428570 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.432420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.432393 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pb7\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.467423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.467389 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:18.509819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.509774 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:14:18.514904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.514881 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555609 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555668 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555705 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555806 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555860 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.555987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.555975 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.556189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.556168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.556275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.556214 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvfx\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.557549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.557520 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.601123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.601084 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:14:18.658455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658419 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658471 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658524 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658671 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658726 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658841 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658889 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.658960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658925 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvfx\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.659067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.658980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.659450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.659413 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.659450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.659427 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.660093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.660037 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.660198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.660153 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.660666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.660645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.662810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.662788 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.662991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.662970 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.726358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.726284 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.738173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.738143 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvfx\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.830297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.830270 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:18.872803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.872766 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:14:18.875754 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:14:18.875696 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4e1a90_f3b3_4d16_a7fb_8ccd08d6b89b.slice/crio-38a8489867151a91c5f4344f460f69c0bf9ae1ea80b5a69b3c6fd82fd7bd1603 WatchSource:0}: Error finding container 38a8489867151a91c5f4344f460f69c0bf9ae1ea80b5a69b3c6fd82fd7bd1603: Status 404 returned error can't find the container with id 38a8489867151a91c5f4344f460f69c0bf9ae1ea80b5a69b3c6fd82fd7bd1603 Oct 30 22:14:18.878112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.878076 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:18.878218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.878166 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:18.878218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:18.878213 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:19.226667 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:14:19.226641 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035687f1_5a2c_4283_94ac_72fb36ea801c.slice/crio-a7ea2e0fb9f2638df6b68413d764048fb2176c279f5a2154c0681f8baaf617f6 WatchSource:0}: Error finding container a7ea2e0fb9f2638df6b68413d764048fb2176c279f5a2154c0681f8baaf617f6: Status 404 returned error can't find the container with id a7ea2e0fb9f2638df6b68413d764048fb2176c279f5a2154c0681f8baaf617f6 Oct 30 22:14:19.226894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.226859 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:14:19.229496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.229460 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:19.229642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.229542 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:19.229642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.229604 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:19.232607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.232559 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" event={"ID":"5d90424d-40d1-4bfd-9780-b3b36b0be4ff","Type":"ContainerStarted","Data":"e25d830a4f8951b4d2608fec8737378b6b2b72c0b93b157abbf34ff4d3749b89"} Oct 30 22:14:19.232707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.232639 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" event={"ID":"5d90424d-40d1-4bfd-9780-b3b36b0be4ff","Type":"ContainerStarted","Data":"01fe347a42b0d6ec21e52aa39eb4e1ffbf19bc6e1e00d2557c6b9bc4e3504676"} Oct 30 22:14:19.234453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.234434 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" event={"ID":"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b","Type":"ContainerStarted","Data":"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e"} Oct 30 22:14:19.234536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.234478 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" event={"ID":"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b","Type":"ContainerStarted","Data":"38a8489867151a91c5f4344f460f69c0bf9ae1ea80b5a69b3c6fd82fd7bd1603"} Oct 30 22:14:19.359243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.359180 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podStartSLOduration=2.359162341 podStartE2EDuration="2.359162341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:19.357363812 +0000 UTC m=+1187.326183377" watchObservedRunningTime="2026-10-30 22:14:19.359162341 +0000 UTC m=+1187.327981908" Oct 30 22:14:19.468658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.468568 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:19.469901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.469872 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:19.470025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.469919 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:19.513001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.512876 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podStartSLOduration=1.5128558079999999 podStartE2EDuration="1.512855808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:19.495802444 +0000 UTC m=+1187.464622009" watchObservedRunningTime="2026-10-30 22:14:19.512855808 +0000 UTC m=+1187.481675373" Oct 30 22:14:19.809747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.809711 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:19.811492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.811461 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:19.811687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:19.811515 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:20.265009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.264550 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" event={"ID":"035687f1-5a2c-4283-94ac-72fb36ea801c","Type":"ContainerStarted","Data":"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5"} Oct 30 22:14:20.265191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.265029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" event={"ID":"035687f1-5a2c-4283-94ac-72fb36ea801c","Type":"ContainerStarted","Data":"a7ea2e0fb9f2638df6b68413d764048fb2176c279f5a2154c0681f8baaf617f6"} Oct 30 22:14:20.468461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.468421 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:20.468635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.468474 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:20.824610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.824363 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:20.824610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.824426 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:20.837910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.837875 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:20.838245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.838221 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:20.838400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:20.838378 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:21.468440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.468399 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:21.468639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.468455 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:21.824140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.824105 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:21.824418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.824389 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:21.836360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.836331 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:21.836916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:21.836888 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:22.468467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.468433 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:22.468670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.468501 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:22.807974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.807936 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:22.808275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.808246 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:22.834227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.834180 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:22.834513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:22.834468 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:23.474600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.469991 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:23.474600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.470070 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:23.821599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.820684 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:23.821599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.820753 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:23.842375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.842340 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:23.842660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:23.842631 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:24.469599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.469381 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:24.469599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.469455 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:24.810600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.809969 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:14:24.810600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.810031 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:14:24.838599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.837985 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:24.838599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:24.838048 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:25.468614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:25.468370 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:14:25.468614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:25.468421 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:14:25.830926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:25.830880 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:14:25.831467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:25.830953 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:14:26.000452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.000412 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:26.000900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.000874 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:26.001849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.001828 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:14:26.195010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.194894 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podStartSLOduration=8.194875776 podStartE2EDuration="8.194875776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:20.403700184 +0000 UTC m=+1188.372519751" watchObservedRunningTime="2026-10-30 22:14:26.194875776 +0000 UTC m=+1194.163695340" Oct 30 22:14:26.640766 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:14:26.640711 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:14:26.643942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.643917 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:26.645019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.644353 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:26.646923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.646899 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:14:26.835283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.835251 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:26.835649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.835515 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:26.836120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:14:26.836099 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:14:56.640099 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:14:56.640007 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:15:00.239659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.239622 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:00.244827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.244799 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:00.244968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.244912 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.249292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.249270 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:00.249400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.249374 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.253397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.253373 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.289901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.289877 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:00.293751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.293734 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.301592 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.301555 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:00.305088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.305069 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.307292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.307271 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.308334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.308311 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:00.336549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.336530 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:15:00.354304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.354269 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:00.359097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359066 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9mb\" (UniqueName: \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.359367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359347 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7bq\" (UniqueName: \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.359551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359528 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.359730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359704 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.359870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359848 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.360041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.359887 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qhb\" (UniqueName: \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.360775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.360752 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:00.404711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.404683 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.406142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.406122 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:00.421861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.421836 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:00.461057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461030 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fmm\" (UniqueName: \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.461172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.461172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461145 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9mb\" (UniqueName: \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.461244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461190 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7bq\" (UniqueName: \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.461244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461221 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.461244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.461395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461262 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.461395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qhb\" (UniqueName: \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.461395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461301 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.461548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.461396 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjptt\" (UniqueName: \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.462059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.462035 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.462059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.462044 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.462172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.462095 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.525763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.525709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9mb\" (UniqueName: \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.528270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.528244 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7bq\" (UniqueName: \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.528360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.528255 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qhb\" (UniqueName: \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.560773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.560755 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:00.562119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.562095 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjptt\" (UniqueName: \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.562206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.562175 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fmm\" (UniqueName: \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.562206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.562198 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.562366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.562347 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.563037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.563016 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.563112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.563096 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.569011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.568992 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:00.575094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.575076 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:00.638366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.636925 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjptt\" (UniqueName: \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\") pod \"maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.641011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.640897 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fmm\" (UniqueName: \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\") pod \"maas-api-key-cleanup-29889975-bzrs5\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.875480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.875448 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz"] Oct 30 22:15:00.880908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.880885 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:00.908989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.908959 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:00.922153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.922130 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:00.937182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.937144 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:00.952951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.952867 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz"] Oct 30 22:15:00.968664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:00.968636 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:01.016602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.015762 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:01.036618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.030059 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:01.070365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.070338 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.070482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.070386 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.070558 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.070499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptlc\" (UniqueName: \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.171588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.171494 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.171758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.171562 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.171758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.171670 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptlc\" (UniqueName: \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.172286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.172261 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.188869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.188844 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.235290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.235259 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptlc\" (UniqueName: \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\") pod \"maas-api-revoke-sub-a12a4058-9wzwz\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.309062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.309034 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:01.334743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.334716 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:01.497824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.497800 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:01.579266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.579230 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerStarted","Data":"07548966d0de19925898477522208c017e0506051504e6ecde31b7994a3830d3"} Oct 30 22:15:01.579392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.579297 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerStarted","Data":"6ad2238964e6cecfa9d9e45a0d9d02ce08cb1820f1089cb1bdb666b32f1529c1"} Oct 30 22:15:01.581332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.581308 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerStarted","Data":"6f0d70553f2de143604572bbc65b8e075f80576747352f21419d4b6dbd08cb3d"} Oct 30 22:15:01.581500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.581355 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerStarted","Data":"aa9879a0f3f9e56c4d5fd3b1f5a00fed85cb436b2a13a4b9f8e8dac64d852cf5"} Oct 30 22:15:01.583305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.583087 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerStarted","Data":"f683ffd706b75cd7d7d1a8aeef157ef3a888c872d92b142a9a6a1bad89296685"} Oct 30 22:15:01.583305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.583135 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerStarted","Data":"504a99cd3d07f9cf3f436bdc69f927792ab6d139083f8b2e775512d1e0504ceb"} Oct 30 22:15:01.584354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.584327 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerStarted","Data":"17d269fa9bafada62e1563cc66cc5951fa60cc9751c5b96dcb0428e09e8d7f78"} Oct 30 22:15:01.585736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.585716 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerStarted","Data":"d322fd25a1eb8112565eeae87f301a0f4857f991484a511512e041d87881ecff"} Oct 30 22:15:01.700815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.700771 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" podStartSLOduration=1.700758391 podStartE2EDuration="1.700758391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.698109179 +0000 UTC m=+1229.666928747" watchObservedRunningTime="2026-10-30 22:15:01.700758391 +0000 UTC m=+1229.669577955" Oct 30 22:15:01.821364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.821319 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" podStartSLOduration=1.821306213 podStartE2EDuration="1.821306213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.818968343 +0000 UTC m=+1229.787787890" watchObservedRunningTime="2026-10-30 22:15:01.821306213 +0000 UTC m=+1229.790125774" Oct 30 22:15:01.886074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.886045 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz"] Oct 30 22:15:01.896131 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:15:01.896104 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f745c6f_c609_4c82_b742_7a6afe3c52df.slice/crio-cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3 WatchSource:0}: Error finding container cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3: Status 404 returned error can't find the container with id cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3 Oct 30 22:15:01.947211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:01.947159 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" podStartSLOduration=1.9471432819999999 podStartE2EDuration="1.947143282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.945347743 +0000 UTC m=+1229.914167308" watchObservedRunningTime="2026-10-30 22:15:01.947143282 +0000 UTC m=+1229.915962845" Oct 30 22:15:02.593986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.593943 4292 generic.go:356] "Generic (PLEG): container finished" podID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerID="6f0d70553f2de143604572bbc65b8e075f80576747352f21419d4b6dbd08cb3d" exitCode=60 Oct 30 22:15:02.594455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.594021 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerDied","Data":"6f0d70553f2de143604572bbc65b8e075f80576747352f21419d4b6dbd08cb3d"} Oct 30 22:15:02.594455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.594305 4292 scope.go:122] "RemoveContainer" containerID="6f0d70553f2de143604572bbc65b8e075f80576747352f21419d4b6dbd08cb3d" Oct 30 22:15:02.595725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.595697 4292 generic.go:356] "Generic (PLEG): container finished" podID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerID="f683ffd706b75cd7d7d1a8aeef157ef3a888c872d92b142a9a6a1bad89296685" exitCode=60 Oct 30 22:15:02.596059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.595770 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerDied","Data":"f683ffd706b75cd7d7d1a8aeef157ef3a888c872d92b142a9a6a1bad89296685"} Oct 30 22:15:02.596059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.595959 4292 scope.go:122] "RemoveContainer" containerID="f683ffd706b75cd7d7d1a8aeef157ef3a888c872d92b142a9a6a1bad89296685" Oct 30 22:15:02.597420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.597364 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" event={"ID":"8f745c6f-c609-4c82-b742-7a6afe3c52df","Type":"ContainerStarted","Data":"371a4f7ac8fcd83aae7d7333ce7e5a5f377461410206208e7e4c9edee9f6aaa4"} Oct 30 22:15:02.597420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.597410 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" event={"ID":"8f745c6f-c609-4c82-b742-7a6afe3c52df","Type":"ContainerStarted","Data":"cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3"} Oct 30 22:15:02.599056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.599038 4292 generic.go:356] "Generic (PLEG): container finished" podID="635fcddc-1580-455b-9522-85416fe4497e" containerID="0833afb2ac90f5cff3b5cd952eeb4d6e8d3ca4be963089f39d15de81e34cb626" exitCode=60 Oct 30 22:15:02.599151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.599127 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerDied","Data":"0833afb2ac90f5cff3b5cd952eeb4d6e8d3ca4be963089f39d15de81e34cb626"} Oct 30 22:15:02.599306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.599281 4292 scope.go:122] "RemoveContainer" containerID="0833afb2ac90f5cff3b5cd952eeb4d6e8d3ca4be963089f39d15de81e34cb626" Oct 30 22:15:02.600656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.600637 4292 generic.go:356] "Generic (PLEG): container finished" podID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerID="4f735fc539d1c16898a594f82968ebc8a1eca4343b6402612b99f0909ca20d07" exitCode=60 Oct 30 22:15:02.600758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.600717 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerDied","Data":"4f735fc539d1c16898a594f82968ebc8a1eca4343b6402612b99f0909ca20d07"} Oct 30 22:15:02.600903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.600887 4292 scope.go:122] "RemoveContainer" containerID="4f735fc539d1c16898a594f82968ebc8a1eca4343b6402612b99f0909ca20d07" Oct 30 22:15:02.602233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.602215 4292 generic.go:356] "Generic (PLEG): container finished" podID="2699d119-287f-48d0-a472-b932bafcc692" containerID="07548966d0de19925898477522208c017e0506051504e6ecde31b7994a3830d3" exitCode=60 Oct 30 22:15:02.602321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.602298 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerDied","Data":"07548966d0de19925898477522208c017e0506051504e6ecde31b7994a3830d3"} Oct 30 22:15:02.602429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:02.602417 4292 scope.go:122] "RemoveContainer" containerID="07548966d0de19925898477522208c017e0506051504e6ecde31b7994a3830d3" Oct 30 22:15:03.209839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.209791 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" podStartSLOduration=3.209776157 podStartE2EDuration="3.209776157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.208724028 +0000 UTC m=+1231.177543590" watchObservedRunningTime="2026-10-30 22:15:03.209776157 +0000 UTC m=+1231.178595719" Oct 30 22:15:03.613533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.612896 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerStarted","Data":"8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd"} Oct 30 22:15:03.615178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.615145 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerStarted","Data":"09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264"} Oct 30 22:15:03.616680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.616651 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f745c6f-c609-4c82-b742-7a6afe3c52df" containerID="371a4f7ac8fcd83aae7d7333ce7e5a5f377461410206208e7e4c9edee9f6aaa4" exitCode=0 Oct 30 22:15:03.616787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.616723 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" event={"ID":"8f745c6f-c609-4c82-b742-7a6afe3c52df","Type":"ContainerDied","Data":"371a4f7ac8fcd83aae7d7333ce7e5a5f377461410206208e7e4c9edee9f6aaa4"} Oct 30 22:15:03.618643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.618618 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerStarted","Data":"6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330"} Oct 30 22:15:03.620505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.620479 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerStarted","Data":"cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4"} Oct 30 22:15:03.622482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.622462 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerStarted","Data":"e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64"} Oct 30 22:15:03.879747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:03.879701 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" podStartSLOduration=3.879687478 podStartE2EDuration="3.879687478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.878492747 +0000 UTC m=+1231.847312308" watchObservedRunningTime="2026-10-30 22:15:03.879687478 +0000 UTC m=+1231.848507041" Oct 30 22:15:04.002214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.002162 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" podStartSLOduration=4.002148749 podStartE2EDuration="4.002148749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.000650366 +0000 UTC m=+1231.969469950" watchObservedRunningTime="2026-10-30 22:15:04.002148749 +0000 UTC m=+1231.970968314" Oct 30 22:15:04.630887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.630847 4292 generic.go:356] "Generic (PLEG): container finished" podID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerID="cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4" exitCode=60 Oct 30 22:15:04.631349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.630932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerDied","Data":"cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4"} Oct 30 22:15:04.631349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.631027 4292 scope.go:122] "RemoveContainer" containerID="4f735fc539d1c16898a594f82968ebc8a1eca4343b6402612b99f0909ca20d07" Oct 30 22:15:04.631349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.631137 4292 scope.go:122] "RemoveContainer" containerID="cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4" Oct 30 22:15:04.631508 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:04.631417 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw_odh-ai-gateway-infra(6dd9b0c2-4add-4012-89ed-58046d3bbd02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" Oct 30 22:15:04.633107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.633085 4292 generic.go:356] "Generic (PLEG): container finished" podID="2699d119-287f-48d0-a472-b932bafcc692" containerID="e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64" exitCode=60 Oct 30 22:15:04.633227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.633164 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerDied","Data":"e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64"} Oct 30 22:15:04.633356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.633342 4292 scope.go:122] "RemoveContainer" containerID="e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64" Oct 30 22:15:04.633587 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:04.633558 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b_odh-ai-gateway-infra(2699d119-287f-48d0-a472-b932bafcc692)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" podUID="2699d119-287f-48d0-a472-b932bafcc692" Oct 30 22:15:04.635059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.635040 4292 generic.go:356] "Generic (PLEG): container finished" podID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerID="8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd" exitCode=60 Oct 30 22:15:04.635164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.635119 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerDied","Data":"8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd"} Oct 30 22:15:04.635293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.635276 4292 scope.go:122] "RemoveContainer" containerID="8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd" Oct 30 22:15:04.635524 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:04.635501 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9_odh-ai-gateway-infra(493a84d6-23cf-4a13-99a4-039f07c22d58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" Oct 30 22:15:04.636964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.636945 4292 generic.go:356] "Generic (PLEG): container finished" podID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerID="09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264" exitCode=60 Oct 30 22:15:04.637051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.637027 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerDied","Data":"09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264"} Oct 30 22:15:04.637206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.637191 4292 scope.go:122] "RemoveContainer" containerID="09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264" Oct 30 22:15:04.637388 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:04.637372 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9_odh-ai-gateway-infra(16ce2c0b-e9fa-4e9d-a55c-a150419000b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" Oct 30 22:15:04.638738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.638720 4292 generic.go:356] "Generic (PLEG): container finished" podID="635fcddc-1580-455b-9522-85416fe4497e" containerID="6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330" exitCode=60 Oct 30 22:15:04.638827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.638807 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerDied","Data":"6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330"} Oct 30 22:15:04.638981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.638965 4292 scope.go:122] "RemoveContainer" containerID="6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330" Oct 30 22:15:04.639202 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:04.639157 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-bzrs5_odh-ai-gateway-infra(635fcddc-1580-455b-9522-85416fe4497e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" podUID="635fcddc-1580-455b-9522-85416fe4497e" Oct 30 22:15:04.645696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.645666 4292 scope.go:122] "RemoveContainer" containerID="07548966d0de19925898477522208c017e0506051504e6ecde31b7994a3830d3" Oct 30 22:15:04.659519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.659502 4292 scope.go:122] "RemoveContainer" containerID="6f0d70553f2de143604572bbc65b8e075f80576747352f21419d4b6dbd08cb3d" Oct 30 22:15:04.673820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.673797 4292 scope.go:122] "RemoveContainer" containerID="f683ffd706b75cd7d7d1a8aeef157ef3a888c872d92b142a9a6a1bad89296685" Oct 30 22:15:04.686982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.686963 4292 scope.go:122] "RemoveContainer" containerID="0833afb2ac90f5cff3b5cd952eeb4d6e8d3ca4be963089f39d15de81e34cb626" Oct 30 22:15:04.804047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.804023 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:04.820868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.820839 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\") pod \"8f745c6f-c609-4c82-b742-7a6afe3c52df\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " Oct 30 22:15:04.820999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.820933 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\") pod \"8f745c6f-c609-4c82-b742-7a6afe3c52df\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " Oct 30 22:15:04.821100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.821086 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\" (UniqueName: \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\") pod \"8f745c6f-c609-4c82-b742-7a6afe3c52df\" (UID: \"8f745c6f-c609-4c82-b742-7a6afe3c52df\") " Oct 30 22:15:04.821326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.821298 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca" pod "8f745c6f-c609-4c82-b742-7a6afe3c52df" (UID: "8f745c6f-c609-4c82-b742-7a6afe3c52df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:04.821565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.821550 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.822856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.822831 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca" pod "8f745c6f-c609-4c82-b742-7a6afe3c52df" (UID: "8f745c6f-c609-4c82-b742-7a6afe3c52df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:04.822978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.822958 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc" pod "8f745c6f-c609-4c82-b742-7a6afe3c52df" (UID: "8f745c6f-c609-4c82-b742-7a6afe3c52df"). InnerVolumeSpecName "kube-api-access-5ptlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:04.922694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.922632 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptlc\" (UniqueName: \"kubernetes.io/projected/8f745c6f-c609-4c82-b742-7a6afe3c52df-kube-api-access-5ptlc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.922694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:04.922654 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f745c6f-c609-4c82-b742-7a6afe3c52df-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:05.648288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.648251 4292 scope.go:122] "RemoveContainer" containerID="cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4" Oct 30 22:15:05.648717 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:05.648537 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw_odh-ai-gateway-infra(6dd9b0c2-4add-4012-89ed-58046d3bbd02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" Oct 30 22:15:05.650012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.649994 4292 scope.go:122] "RemoveContainer" containerID="e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64" Oct 30 22:15:05.650228 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:05.650202 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b_odh-ai-gateway-infra(2699d119-287f-48d0-a472-b932bafcc692)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" podUID="2699d119-287f-48d0-a472-b932bafcc692" Oct 30 22:15:05.651775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.651756 4292 scope.go:122] "RemoveContainer" containerID="8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd" Oct 30 22:15:05.652002 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:05.651982 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9_odh-ai-gateway-infra(493a84d6-23cf-4a13-99a4-039f07c22d58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" Oct 30 22:15:05.653257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.653239 4292 scope.go:122] "RemoveContainer" containerID="09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264" Oct 30 22:15:05.653449 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:05.653430 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9_odh-ai-gateway-infra(16ce2c0b-e9fa-4e9d-a55c-a150419000b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" Oct 30 22:15:05.654268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.654252 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" Oct 30 22:15:05.654336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.654271 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz" event={"ID":"8f745c6f-c609-4c82-b742-7a6afe3c52df","Type":"ContainerDied","Data":"cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3"} Oct 30 22:15:05.654336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.654312 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb0cbc574c3788d0083998ea9098745fadfe540fed6b33067685a4e570c04df3" Oct 30 22:15:05.655982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:05.655966 4292 scope.go:122] "RemoveContainer" containerID="6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330" Oct 30 22:15:05.656154 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:05.656137 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-bzrs5_odh-ai-gateway-infra(635fcddc-1580-455b-9522-85416fe4497e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" podUID="635fcddc-1580-455b-9522-85416fe4497e" Oct 30 22:15:14.139478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:14.139445 4292 scope.go:122] "RemoveContainer" containerID="8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd" Oct 30 22:15:14.792556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:14.792479 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerStarted","Data":"d7f21856a573ae70bfbf18d2ba223821fd0eefa42c0ab2ade94a92f18c8649ed"} Oct 30 22:15:15.139633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.139590 4292 scope.go:122] "RemoveContainer" containerID="cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4" Oct 30 22:15:15.140090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.139674 4292 scope.go:122] "RemoveContainer" containerID="6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330" Oct 30 22:15:15.802103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.802074 4292 generic.go:356] "Generic (PLEG): container finished" podID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerID="d7f21856a573ae70bfbf18d2ba223821fd0eefa42c0ab2ade94a92f18c8649ed" exitCode=60 Oct 30 22:15:15.802270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.802152 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerDied","Data":"d7f21856a573ae70bfbf18d2ba223821fd0eefa42c0ab2ade94a92f18c8649ed"} Oct 30 22:15:15.802270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.802246 4292 scope.go:122] "RemoveContainer" containerID="8d43c1d6cc7541c19e0b00b57d0fc242e83c7571ec3df884a9e25b73758a62bd" Oct 30 22:15:15.802636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.802612 4292 scope.go:122] "RemoveContainer" containerID="d7f21856a573ae70bfbf18d2ba223821fd0eefa42c0ab2ade94a92f18c8649ed" Oct 30 22:15:15.802904 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:15.802878 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9_odh-ai-gateway-infra(493a84d6-23cf-4a13-99a4-039f07c22d58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" Oct 30 22:15:15.804361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.804337 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerStarted","Data":"7ea48bc82286c2f22f6f7033bec511cae8401e4233ade7be7924818ed05b4a22"} Oct 30 22:15:15.806204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.806178 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerStarted","Data":"7c6ca4c40c853c734550645e297a687922fafddf58447a71b215a5d5bb923b82"} Oct 30 22:15:15.967745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:15.967719 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:16.819063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.818860 4292 generic.go:356] "Generic (PLEG): container finished" podID="635fcddc-1580-455b-9522-85416fe4497e" containerID="7ea48bc82286c2f22f6f7033bec511cae8401e4233ade7be7924818ed05b4a22" exitCode=60 Oct 30 22:15:16.819063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.819014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerDied","Data":"7ea48bc82286c2f22f6f7033bec511cae8401e4233ade7be7924818ed05b4a22"} Oct 30 22:15:16.819529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.819124 4292 scope.go:122] "RemoveContainer" containerID="6eb081dee3e2a00a0830f7edd1e1f0f9990f794b9a4890dd03599eadefc60330" Oct 30 22:15:16.819636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.819537 4292 scope.go:122] "RemoveContainer" containerID="7ea48bc82286c2f22f6f7033bec511cae8401e4233ade7be7924818ed05b4a22" Oct 30 22:15:16.819916 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:16.819861 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-bzrs5_odh-ai-gateway-infra(635fcddc-1580-455b-9522-85416fe4497e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" podUID="635fcddc-1580-455b-9522-85416fe4497e" Oct 30 22:15:16.821804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.821780 4292 generic.go:356] "Generic (PLEG): container finished" podID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerID="7c6ca4c40c853c734550645e297a687922fafddf58447a71b215a5d5bb923b82" exitCode=60 Oct 30 22:15:16.821910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.821826 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerDied","Data":"7c6ca4c40c853c734550645e297a687922fafddf58447a71b215a5d5bb923b82"} Oct 30 22:15:16.822190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.822176 4292 scope.go:122] "RemoveContainer" containerID="7c6ca4c40c853c734550645e297a687922fafddf58447a71b215a5d5bb923b82" Oct 30 22:15:16.822487 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:16.822468 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw_odh-ai-gateway-infra(6dd9b0c2-4add-4012-89ed-58046d3bbd02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" Oct 30 22:15:16.838378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.836006 4292 scope.go:122] "RemoveContainer" containerID="cc6d2a592f3e0887face8730d2ec254e00c10f19808950bcff70551f1fa134a4" Oct 30 22:15:16.981971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:16.981948 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:17.062042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.062012 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\" (UniqueName: \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\") pod \"493a84d6-23cf-4a13-99a4-039f07c22d58\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " Oct 30 22:15:17.062179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.062134 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\") pod \"493a84d6-23cf-4a13-99a4-039f07c22d58\" (UID: \"493a84d6-23cf-4a13-99a4-039f07c22d58\") " Oct 30 22:15:17.062420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.062392 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca" pod "493a84d6-23cf-4a13-99a4-039f07c22d58" (UID: "493a84d6-23cf-4a13-99a4-039f07c22d58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:17.062562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.062546 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a84d6-23cf-4a13-99a4-039f07c22d58-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.064088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.064056 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb" pod "493a84d6-23cf-4a13-99a4-039f07c22d58" (UID: "493a84d6-23cf-4a13-99a4-039f07c22d58"). InnerVolumeSpecName "kube-api-access-5l9mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:17.163938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.163908 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9mb\" (UniqueName: \"kubernetes.io/projected/493a84d6-23cf-4a13-99a4-039f07c22d58-kube-api-access-5l9mb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.274719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.274690 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:17.324215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.324137 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:17.834161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.834129 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" event={"ID":"493a84d6-23cf-4a13-99a4-039f07c22d58","Type":"ContainerDied","Data":"aa9879a0f3f9e56c4d5fd3b1f5a00fed85cb436b2a13a4b9f8e8dac64d852cf5"} Oct 30 22:15:17.834675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.834134 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9" Oct 30 22:15:17.834675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:17.834656 4292 scope.go:122] "RemoveContainer" containerID="d7f21856a573ae70bfbf18d2ba223821fd0eefa42c0ab2ade94a92f18c8649ed" Oct 30 22:15:18.020522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.020495 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:18.048534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.048512 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:18.093844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.093766 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:18.099763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.099738 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889975-v29n9"] Oct 30 22:15:18.140152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.140125 4292 scope.go:122] "RemoveContainer" containerID="e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64" Oct 30 22:15:18.140259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.140216 4292 scope.go:122] "RemoveContainer" containerID="09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264" Oct 30 22:15:18.147970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.147949 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" path="/var/lib/kubelet/pods/493a84d6-23cf-4a13-99a4-039f07c22d58/volumes" Oct 30 22:15:18.177548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.177523 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\" (UniqueName: \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\") pod \"635fcddc-1580-455b-9522-85416fe4497e\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " Oct 30 22:15:18.177726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.177709 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\") pod \"635fcddc-1580-455b-9522-85416fe4497e\" (UID: \"635fcddc-1580-455b-9522-85416fe4497e\") " Oct 30 22:15:18.177768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.177734 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\") pod \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " Oct 30 22:15:18.177807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.177787 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\" (UniqueName: \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\") pod \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\" (UID: \"6dd9b0c2-4add-4012-89ed-58046d3bbd02\") " Oct 30 22:15:18.178119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.178098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca" pod "635fcddc-1580-455b-9522-85416fe4497e" (UID: "635fcddc-1580-455b-9522-85416fe4497e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.178183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.178098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca" pod "6dd9b0c2-4add-4012-89ed-58046d3bbd02" (UID: "6dd9b0c2-4add-4012-89ed-58046d3bbd02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.178223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.178215 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635fcddc-1580-455b-9522-85416fe4497e-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.178258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.178227 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9b0c2-4add-4012-89ed-58046d3bbd02-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.179469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.179445 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm" pod "635fcddc-1580-455b-9522-85416fe4497e" (UID: "635fcddc-1580-455b-9522-85416fe4497e"). InnerVolumeSpecName "kube-api-access-76fmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.179697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.179675 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt" pod "6dd9b0c2-4add-4012-89ed-58046d3bbd02" (UID: "6dd9b0c2-4add-4012-89ed-58046d3bbd02"). InnerVolumeSpecName "kube-api-access-qjptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.279736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.279706 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjptt\" (UniqueName: \"kubernetes.io/projected/6dd9b0c2-4add-4012-89ed-58046d3bbd02-kube-api-access-qjptt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.279736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.279729 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fmm\" (UniqueName: \"kubernetes.io/projected/635fcddc-1580-455b-9522-85416fe4497e-kube-api-access-76fmm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.845170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.845144 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" Oct 30 22:15:18.845563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.845177 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw" event={"ID":"6dd9b0c2-4add-4012-89ed-58046d3bbd02","Type":"ContainerDied","Data":"d322fd25a1eb8112565eeae87f301a0f4857f991484a511512e041d87881ecff"} Oct 30 22:15:18.845563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.845251 4292 scope.go:122] "RemoveContainer" containerID="7c6ca4c40c853c734550645e297a687922fafddf58447a71b215a5d5bb923b82" Oct 30 22:15:18.847628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.847557 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerStarted","Data":"101ca77a4665721607f0b6d106fe54b95ea49819ffbd4a71b688c5e9e9d8fd0c"} Oct 30 22:15:18.850849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.850821 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerStarted","Data":"b3b97be638046da10a089f324f77415bf9da8265c43cb8a39a139971ac4a4703"} Oct 30 22:15:18.852417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.852401 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" Oct 30 22:15:18.852417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.852418 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5" event={"ID":"635fcddc-1580-455b-9522-85416fe4497e","Type":"ContainerDied","Data":"17d269fa9bafada62e1563cc66cc5951fa60cc9751c5b96dcb0428e09e8d7f78"} Oct 30 22:15:18.856219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:18.856204 4292 scope.go:122] "RemoveContainer" containerID="7ea48bc82286c2f22f6f7033bec511cae8401e4233ade7be7924818ed05b4a22" Oct 30 22:15:19.017876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.017851 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:19.024775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.024750 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-075cbb-29889975-7qscw"] Oct 30 22:15:19.418282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.418194 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:19.428596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.428552 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-bzrs5"] Oct 30 22:15:19.863235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.863196 4292 generic.go:356] "Generic (PLEG): container finished" podID="2699d119-287f-48d0-a472-b932bafcc692" containerID="101ca77a4665721607f0b6d106fe54b95ea49819ffbd4a71b688c5e9e9d8fd0c" exitCode=60 Oct 30 22:15:19.863235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.863225 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerDied","Data":"101ca77a4665721607f0b6d106fe54b95ea49819ffbd4a71b688c5e9e9d8fd0c"} Oct 30 22:15:19.863771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.863282 4292 scope.go:122] "RemoveContainer" containerID="e94d224c7a61cda02b6d8250ca383bda787edc7c0543ed67a6f07947c6bbcd64" Oct 30 22:15:19.863771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.863687 4292 scope.go:122] "RemoveContainer" containerID="101ca77a4665721607f0b6d106fe54b95ea49819ffbd4a71b688c5e9e9d8fd0c" Oct 30 22:15:19.864039 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:19.864010 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b_odh-ai-gateway-infra(2699d119-287f-48d0-a472-b932bafcc692)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" podUID="2699d119-287f-48d0-a472-b932bafcc692" Oct 30 22:15:19.865327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.865302 4292 generic.go:356] "Generic (PLEG): container finished" podID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerID="b3b97be638046da10a089f324f77415bf9da8265c43cb8a39a139971ac4a4703" exitCode=60 Oct 30 22:15:19.865438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.865346 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerDied","Data":"b3b97be638046da10a089f324f77415bf9da8265c43cb8a39a139971ac4a4703"} Oct 30 22:15:19.865707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.865687 4292 scope.go:122] "RemoveContainer" containerID="b3b97be638046da10a089f324f77415bf9da8265c43cb8a39a139971ac4a4703" Oct 30 22:15:19.865978 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:19.865952 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9_odh-ai-gateway-infra(16ce2c0b-e9fa-4e9d-a55c-a150419000b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" Oct 30 22:15:19.876201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:19.876184 4292 scope.go:122] "RemoveContainer" containerID="09c7099b6a7445c46df1f762a8e0245e74bd803df27ee0677ab3e7a07f0cd264" Oct 30 22:15:20.147189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:20.147113 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635fcddc-1580-455b-9522-85416fe4497e" path="/var/lib/kubelet/pods/635fcddc-1580-455b-9522-85416fe4497e/volumes" Oct 30 22:15:20.147531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:20.147517 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" path="/var/lib/kubelet/pods/6dd9b0c2-4add-4012-89ed-58046d3bbd02/volumes" Oct 30 22:15:20.167725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:20.167694 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:20.295674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:20.295637 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:21.027681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.027646 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:21.064538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.064510 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:21.115413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.115389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\") pod \"2699d119-287f-48d0-a472-b932bafcc692\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " Oct 30 22:15:21.115504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.115484 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\" (UniqueName: \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\") pod \"2699d119-287f-48d0-a472-b932bafcc692\" (UID: \"2699d119-287f-48d0-a472-b932bafcc692\") " Oct 30 22:15:21.115739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.115721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca" pod "2699d119-287f-48d0-a472-b932bafcc692" (UID: "2699d119-287f-48d0-a472-b932bafcc692"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.115898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.115881 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2699d119-287f-48d0-a472-b932bafcc692-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.117959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.117937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb" pod "2699d119-287f-48d0-a472-b932bafcc692" (UID: "2699d119-287f-48d0-a472-b932bafcc692"). InnerVolumeSpecName "kube-api-access-k5qhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.216881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.216850 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\" (UniqueName: \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\") pod \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " Oct 30 22:15:21.217033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.217018 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\") pod \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\" (UID: \"16ce2c0b-e9fa-4e9d-a55c-a150419000b4\") " Oct 30 22:15:21.217410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.217381 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca" pod "16ce2c0b-e9fa-4e9d-a55c-a150419000b4" (UID: "16ce2c0b-e9fa-4e9d-a55c-a150419000b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.217550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.217533 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.217612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.217553 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5qhb\" (UniqueName: \"kubernetes.io/projected/2699d119-287f-48d0-a472-b932bafcc692-kube-api-access-k5qhb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.218859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.218843 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq" pod "16ce2c0b-e9fa-4e9d-a55c-a150419000b4" (UID: "16ce2c0b-e9fa-4e9d-a55c-a150419000b4"). InnerVolumeSpecName "kube-api-access-jv7bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.319075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.319041 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv7bq\" (UniqueName: \"kubernetes.io/projected/16ce2c0b-e9fa-4e9d-a55c-a150419000b4-kube-api-access-jv7bq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.885023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.884992 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" Oct 30 22:15:21.885182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.885026 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b" event={"ID":"2699d119-287f-48d0-a472-b932bafcc692","Type":"ContainerDied","Data":"6ad2238964e6cecfa9d9e45a0d9d02ce08cb1820f1089cb1bdb666b32f1529c1"} Oct 30 22:15:21.885182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.885091 4292 scope.go:122] "RemoveContainer" containerID="101ca77a4665721607f0b6d106fe54b95ea49819ffbd4a71b688c5e9e9d8fd0c" Oct 30 22:15:21.886721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.886697 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" Oct 30 22:15:21.886889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.886748 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9" event={"ID":"16ce2c0b-e9fa-4e9d-a55c-a150419000b4","Type":"ContainerDied","Data":"504a99cd3d07f9cf3f436bdc69f927792ab6d139083f8b2e775512d1e0504ceb"} Oct 30 22:15:21.899377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:21.899358 4292 scope.go:122] "RemoveContainer" containerID="b3b97be638046da10a089f324f77415bf9da8265c43cb8a39a139971ac4a4703" Oct 30 22:15:22.060282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:22.060252 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:22.066796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:22.066773 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889975-rhxn9"] Oct 30 22:15:22.146683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:22.146591 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" path="/var/lib/kubelet/pods/16ce2c0b-e9fa-4e9d-a55c-a150419000b4/volumes" Oct 30 22:15:22.181979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:22.181950 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:22.190130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:22.190103 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889975-nm47b"] Oct 30 22:15:24.147688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:24.147654 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2699d119-287f-48d0-a472-b932bafcc692" path="/var/lib/kubelet/pods/2699d119-287f-48d0-a472-b932bafcc692/volumes" Oct 30 22:15:26.639217 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:26.639184 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:15:33.171600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.171551 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r"] Oct 30 22:15:33.172346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172323 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172348 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172365 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172372 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172382 4292 cpu_manager.go:414] "containerMap: removing container" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172389 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172405 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172413 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172426 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172432 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172446 4292 cpu_manager.go:414] "containerMap: removing container" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.172455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172453 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172468 4292 cpu_manager.go:414] "containerMap: removing container" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172489 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172504 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172512 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172529 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172536 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172565 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172589 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172602 4292 cpu_manager.go:414] "containerMap: removing container" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172610 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172627 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f745c6f-c609-4c82-b742-7a6afe3c52df" containerName="revoke-keys" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172635 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f745c6f-c609-4c82-b742-7a6afe3c52df" containerName="revoke-keys" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172661 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172668 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172812 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172826 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172836 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172849 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f745c6f-c609-4c82-b742-7a6afe3c52df" containerName="revoke-keys" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172859 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172872 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172879 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172888 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172898 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172906 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172915 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172924 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:33.173074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.172934 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2c0b-e9fa-4e9d-a55c-a150419000b4" containerName="cleanup" Oct 30 22:15:33.178525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.178503 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.234643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.234618 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r"] Oct 30 22:15:33.235323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.235302 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:33.261945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.261910 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.262072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.262037 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rj62\" (UniqueName: \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.262139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.262077 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.280163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.280144 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:33.280408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.280390 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:33.285837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.285824 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:15:33.363548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.363523 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.363660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.363646 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rj62\" (UniqueName: \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.363711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.363674 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.364433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.364407 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.365998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.365973 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.429201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.429143 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rj62\" (UniqueName: \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\") pod \"maas-api-revoke-sub-3ceaebf0-qhh5r\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.492397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.492372 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:33.870388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.870360 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r"] Oct 30 22:15:33.871021 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:15:33.870994 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a34412d_8052_47f4_8717_022337f08141.slice/crio-f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0 WatchSource:0}: Error finding container f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0: Status 404 returned error can't find the container with id f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0 Oct 30 22:15:33.967632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:33.967601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" event={"ID":"3a34412d-8052-47f4-8717-022337f08141","Type":"ContainerStarted","Data":"f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0"} Oct 30 22:15:34.975369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:34.975279 4292 generic.go:356] "Generic (PLEG): container finished" podID="3a34412d-8052-47f4-8717-022337f08141" containerID="5445f5f41c73d0ea6d3472aa44452218e145acd90d21c68f156faa60d3d8d7b3" exitCode=0 Oct 30 22:15:34.975801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:34.975369 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" event={"ID":"3a34412d-8052-47f4-8717-022337f08141","Type":"ContainerDied","Data":"5445f5f41c73d0ea6d3472aa44452218e145acd90d21c68f156faa60d3d8d7b3"} Oct 30 22:15:36.133440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.133411 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:36.195478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.195449 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\" (UniqueName: \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\") pod \"3a34412d-8052-47f4-8717-022337f08141\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " Oct 30 22:15:36.195637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.195489 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\") pod \"3a34412d-8052-47f4-8717-022337f08141\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " Oct 30 22:15:36.195637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.195620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\") pod \"3a34412d-8052-47f4-8717-022337f08141\" (UID: \"3a34412d-8052-47f4-8717-022337f08141\") " Oct 30 22:15:36.195984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.195961 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca" pod "3a34412d-8052-47f4-8717-022337f08141" (UID: "3a34412d-8052-47f4-8717-022337f08141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:36.197496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.197470 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca" pod "3a34412d-8052-47f4-8717-022337f08141" (UID: "3a34412d-8052-47f4-8717-022337f08141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:36.197627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.197495 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62" pod "3a34412d-8052-47f4-8717-022337f08141" (UID: "3a34412d-8052-47f4-8717-022337f08141"). InnerVolumeSpecName "kube-api-access-7rj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:36.297332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.297303 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rj62\" (UniqueName: \"kubernetes.io/projected/3a34412d-8052-47f4-8717-022337f08141-kube-api-access-7rj62\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:36.297332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.297331 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34412d-8052-47f4-8717-022337f08141-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:36.297513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.297343 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34412d-8052-47f4-8717-022337f08141-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:36.395898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.395867 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:15:36.396418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396403 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:36.396469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396419 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd9b0c2-4add-4012-89ed-58046d3bbd02" containerName="cleanup" Oct 30 22:15:36.396469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396429 4292 cpu_manager.go:414] "containerMap: removing container" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:36.396469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396434 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:36.396469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396462 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3a34412d-8052-47f4-8717-022337f08141" containerName="revoke-keys" Oct 30 22:15:36.396469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396467 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a34412d-8052-47f4-8717-022337f08141" containerName="revoke-keys" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396476 4292 cpu_manager.go:414] "containerMap: removing container" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396481 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396590 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a84d6-23cf-4a13-99a4-039f07c22d58" containerName="cleanup" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396600 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a34412d-8052-47f4-8717-022337f08141" containerName="revoke-keys" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396608 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699d119-287f-48d0-a472-b932bafcc692" containerName="cleanup" Oct 30 22:15:36.396653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.396620 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="635fcddc-1580-455b-9522-85416fe4497e" containerName="cleanup" Oct 30 22:15:36.400297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.400281 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.456375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.456347 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-075cbb\"/\"distinct-2-w5-075cbb-kserve-self-signed-certs\"" Oct 30 22:15:36.460171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.460143 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:15:36.500199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffhb\" (UniqueName: \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.500327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500209 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.500368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500341 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.500463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500448 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.500498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500489 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.500535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.500509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.601706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.601610 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.601867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.601732 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.601867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.601828 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.601880 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.601906 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.602010 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffhb\" (UniqueName: \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.602146 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.602171 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.602288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.602248 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.604033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.604006 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.604420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.604399 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.665447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.665419 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffhb\" (UniqueName: \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\") pod \"distinct-2-w5-075cbb-kserve-f6685668b-jd97f\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.713145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.713115 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:36.990547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.990476 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" Oct 30 22:15:36.990547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.990515 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r" event={"ID":"3a34412d-8052-47f4-8717-022337f08141","Type":"ContainerDied","Data":"f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0"} Oct 30 22:15:36.990735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:36.990568 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e492313c0161ee193606a8631167820da4749d67859350abc810b3f1a03bc0" Oct 30 22:15:37.092478 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:15:37.092451 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod062d5db4_1a5c_44b8_af2d_005e0a54cf5c.slice/crio-d1f886b06a282d491f391a78c1a08b23dff6bf2ea46fd6972a2b65b223916dfa WatchSource:0}: Error finding container d1f886b06a282d491f391a78c1a08b23dff6bf2ea46fd6972a2b65b223916dfa: Status 404 returned error can't find the container with id d1f886b06a282d491f391a78c1a08b23dff6bf2ea46fd6972a2b65b223916dfa Oct 30 22:15:37.093063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:37.093033 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:15:37.999228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:37.999189 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" event={"ID":"062d5db4-1a5c-44b8-af2d-005e0a54cf5c","Type":"ContainerStarted","Data":"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d"} Oct 30 22:15:37.999228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:37.999235 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" event={"ID":"062d5db4-1a5c-44b8-af2d-005e0a54cf5c","Type":"ContainerStarted","Data":"d1f886b06a282d491f391a78c1a08b23dff6bf2ea46fd6972a2b65b223916dfa"} Oct 30 22:15:38.121683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:38.121636 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podStartSLOduration=2.1216236410000002 podStartE2EDuration="2.121623641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:38.119853347 +0000 UTC m=+1266.088672912" watchObservedRunningTime="2026-10-30 22:15:38.121623641 +0000 UTC m=+1266.090443221" Oct 30 22:15:38.713943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:38.713907 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:44.078416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:44.078376 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm"] Oct 30 22:15:44.129157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:44.129122 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804277aa-tkwhm"] Oct 30 22:15:44.146374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:44.146340 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63603e25-98fc-4282-b5d4-400637c53956" path="/var/lib/kubelet/pods/63603e25-98fc-4282-b5d4-400637c53956/volumes" Oct 30 22:15:46.714083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:46.714048 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:46.727101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:46.727075 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:47.071823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:47.071793 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:15:49.041543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.041506 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8"] Oct 30 22:15:49.046046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.046024 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.104901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.104875 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8"] Oct 30 22:15:49.150295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.150268 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:49.150295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.150275 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:49.150441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.150377 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:49.150441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.150409 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:15:49.166041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.166018 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.166173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.166155 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.166240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.166225 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk688\" (UniqueName: \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.267404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.267375 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk688\" (UniqueName: \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.267519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.267490 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.267669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.267651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.268135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.268112 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.269920 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.269897 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.330278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.330253 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk688\" (UniqueName: \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\") pod \"maas-api-revoke-sub-d9208411-qlvp8\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.359477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.359457 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:49.739465 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:15:49.739431 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc9dd1b_d5ab_490e_8456_5021f9be4d1b.slice/crio-ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c WatchSource:0}: Error finding container ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c: Status 404 returned error can't find the container with id ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c Oct 30 22:15:49.739725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.739703 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8"] Oct 30 22:15:49.934859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.934826 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:15:49.940055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.940040 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:49.995816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:49.995793 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-13b8d8\"/\"distinct-2-w2-13b8d8-kserve-self-signed-certs\"" Oct 30 22:15:50.001298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.001259 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.077997 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs2zz\" (UniqueName: \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.078076 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.078134 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.078181 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.078207 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.078381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.078250 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.100796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.100760 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" event={"ID":"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b","Type":"ContainerStarted","Data":"60132aa4a3291e616a428f227a321adfb39a9e0b4dffdb794a015754560b8808"} Oct 30 22:15:50.101022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.101002 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" event={"ID":"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b","Type":"ContainerStarted","Data":"ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c"} Oct 30 22:15:50.181881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.181849 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs2zz\" (UniqueName: \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.181919 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.181981 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182035 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182063 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182104 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182368 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.182673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.182653 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.184302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.184275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.184754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.184733 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.223054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.223006 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" podStartSLOduration=2.222994339 podStartE2EDuration="2.222994339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:50.221993941 +0000 UTC m=+1278.190813506" watchObservedRunningTime="2026-10-30 22:15:50.222994339 +0000 UTC m=+1278.191813903" Oct 30 22:15:50.245092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.245044 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs2zz\" (UniqueName: \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\") pod \"distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.251902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.251885 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:50.639897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:50.639862 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:15:50.658386 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:15:50.658359 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2da699f_9b5f_41cb_9d5c_83c1aa2f4407.slice/crio-91b918704b7c38ca474c7392d40fd2a4103f8956de92768f09bfef6f787a4d42 WatchSource:0}: Error finding container 91b918704b7c38ca474c7392d40fd2a4103f8956de92768f09bfef6f787a4d42: Status 404 returned error can't find the container with id 91b918704b7c38ca474c7392d40fd2a4103f8956de92768f09bfef6f787a4d42 Oct 30 22:15:51.108796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.108761 4292 generic.go:356] "Generic (PLEG): container finished" podID="dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" containerID="60132aa4a3291e616a428f227a321adfb39a9e0b4dffdb794a015754560b8808" exitCode=0 Oct 30 22:15:51.109222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.108817 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" event={"ID":"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b","Type":"ContainerDied","Data":"60132aa4a3291e616a428f227a321adfb39a9e0b4dffdb794a015754560b8808"} Oct 30 22:15:51.110518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.110496 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" event={"ID":"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407","Type":"ContainerStarted","Data":"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46"} Oct 30 22:15:51.110666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.110529 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" event={"ID":"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407","Type":"ContainerStarted","Data":"91b918704b7c38ca474c7392d40fd2a4103f8956de92768f09bfef6f787a4d42"} Oct 30 22:15:51.252237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.252201 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:15:51.349512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:51.349465 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podStartSLOduration=2.349450976 podStartE2EDuration="2.349450976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:51.347302525 +0000 UTC m=+1279.316122089" watchObservedRunningTime="2026-10-30 22:15:51.349450976 +0000 UTC m=+1279.318270578" Oct 30 22:15:52.256404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.256379 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:52.305423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.305393 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\") pod \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " Oct 30 22:15:52.305553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.305467 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\" (UniqueName: \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\") pod \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " Oct 30 22:15:52.305653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.305613 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\") pod \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\" (UID: \"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b\") " Oct 30 22:15:52.305943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.305921 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca" pod "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" (UID: "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:52.306047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.306031 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:52.307431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.307411 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688" pod "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" (UID: "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b"). InnerVolumeSpecName "kube-api-access-rk688". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:52.307497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.307437 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca" pod "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" (UID: "dcc9dd1b-d5ab-490e-8456-5021f9be4d1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:52.407273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.407205 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:52.407273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:52.407233 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk688\" (UniqueName: \"kubernetes.io/projected/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b-kube-api-access-rk688\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:53.124260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:53.124222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" event={"ID":"dcc9dd1b-d5ab-490e-8456-5021f9be4d1b","Type":"ContainerDied","Data":"ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c"} Oct 30 22:15:53.124260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:53.124245 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8" Oct 30 22:15:53.124464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:53.124277 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca161cb58de14b83669fca61cbc66fa9f0e274560e5d3bf42c305fe946d0465c" Oct 30 22:15:55.057891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:55.057859 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c"] Oct 30 22:15:55.107981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:55.107956 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a370166-skv8c"] Oct 30 22:15:56.147892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:56.147851 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec43f881-19ee-42ba-a2d6-886028e7d0a3" path="/var/lib/kubelet/pods/ec43f881-19ee-42ba-a2d6-886028e7d0a3/volumes" Oct 30 22:15:56.639769 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:15:56.639727 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:15:59.814775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.814742 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:15:59.815357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.815341 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" containerName="revoke-keys" Oct 30 22:15:59.815417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.815358 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" containerName="revoke-keys" Oct 30 22:15:59.815453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.815445 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" containerName="revoke-keys" Oct 30 22:15:59.819169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.819149 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.894908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.894876 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.894920 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7tg\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.894951 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.894970 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895232 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.895183 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.895375 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.895417 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.895500 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.895606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.895566 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.930176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.930144 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:15:59.996771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.996731 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.996928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.996868 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.996928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.996901 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997025 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997098 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997132 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997159 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7tg\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997301 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997348 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997366 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997436 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997650 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.997795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.997775 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.999382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.999358 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:15:59.999700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:15:59.999683 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:00.059523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.059491 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:00.059666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.059627 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7tg\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:00.133050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.132968 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:00.252363 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.252328 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:16:00.265617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.265592 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:16:00.574625 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:00.574567 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029eb867_dcc6_45e5_a2c2_1798192b4f4d.slice/crio-b1435d57c66a1f0ee5b68deb31fe31a66aefe3c57a8a825d02912f40f81c1c06 WatchSource:0}: Error finding container b1435d57c66a1f0ee5b68deb31fe31a66aefe3c57a8a825d02912f40f81c1c06: Status 404 returned error can't find the container with id b1435d57c66a1f0ee5b68deb31fe31a66aefe3c57a8a825d02912f40f81c1c06 Oct 30 22:16:00.576567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.576539 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:16:00.577142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.577111 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:00.577196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.577179 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:00.577228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:00.577209 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:01.245936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:01.236437 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" event={"ID":"029eb867-dcc6-45e5-a2c2-1798192b4f4d","Type":"ContainerStarted","Data":"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4"} Oct 30 22:16:01.245936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:01.236503 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" event={"ID":"029eb867-dcc6-45e5-a2c2-1798192b4f4d","Type":"ContainerStarted","Data":"b1435d57c66a1f0ee5b68deb31fe31a66aefe3c57a8a825d02912f40f81c1c06"} Oct 30 22:16:01.397240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:01.397177 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podStartSLOduration=2.397157133 podStartE2EDuration="2.397157133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:01.394060504 +0000 UTC m=+1289.362880069" watchObservedRunningTime="2026-10-30 22:16:01.397157133 +0000 UTC m=+1289.365976698" Oct 30 22:16:01.599600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:01.598897 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:16:02.171600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:02.168462 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:02.171600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:02.168523 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:02.228600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:02.220832 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:03.147601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:03.136485 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:03.147601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:03.136567 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:04.133615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:04.133555 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:04.133803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:04.133632 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:05.133665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:05.133628 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:05.134089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:05.133681 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:06.133566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:06.133531 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.133776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:06.133627 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:07.133479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:07.133441 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.133665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:07.133516 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:09.066420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.066385 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97"] Oct 30 22:16:09.116226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.116190 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2de42ec-2bg97"] Oct 30 22:16:09.174679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.174652 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:16:09.174840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.174699 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:09.474856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.474765 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:09.475234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.475207 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:09.475912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:09.475894 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:16:10.147668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:10.147627 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1c8dfd-e9b1-4196-972f-38bb21f2479c" path="/var/lib/kubelet/pods/3e1c8dfd-e9b1-4196-972f-38bb21f2479c/volumes" Oct 30 22:16:12.058902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:12.058866 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj"] Oct 30 22:16:12.109147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:12.109114 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1033e88-cf2mj"] Oct 30 22:16:12.155595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:12.155534 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326fcb1f-29b9-49c7-aec1-d2977b75eeee" path="/var/lib/kubelet/pods/326fcb1f-29b9-49c7-aec1-d2977b75eeee/volumes" Oct 30 22:16:13.631352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.631317 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94"] Oct 30 22:16:13.636612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.636594 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.695617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.695586 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94"] Oct 30 22:16:13.739948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.739912 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65fk\" (UniqueName: \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.740109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.739987 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.740109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.740041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.741491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.741474 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:13.741763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.741743 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:13.741830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.741748 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:16:13.741830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.741752 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:13.841452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.841429 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65fk\" (UniqueName: \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.841565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.841484 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.841565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.841513 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.842205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.842179 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.843808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.843790 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.905357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.905298 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65fk\" (UniqueName: \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\") pod \"maas-api-revoke-sub-b3b85fe4-d5s94\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:13.950338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:13.950312 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:14.327763 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:14.327729 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f571ad_c298_4ce8_a013_92964a4b71d6.slice/crio-c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f WatchSource:0}: Error finding container c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f: Status 404 returned error can't find the container with id c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f Oct 30 22:16:14.327906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:14.327888 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94"] Oct 30 22:16:14.383737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:14.383708 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" event={"ID":"b3f571ad-c298-4ce8-a013-92964a4b71d6","Type":"ContainerStarted","Data":"c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f"} Oct 30 22:16:15.391753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:15.391716 4292 generic.go:356] "Generic (PLEG): container finished" podID="b3f571ad-c298-4ce8-a013-92964a4b71d6" containerID="8a02a7daac6c4a7a648f47e7fa5203884e806e7b2f201de5acb089aa0085efd9" exitCode=0 Oct 30 22:16:15.392146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:15.391769 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" event={"ID":"b3f571ad-c298-4ce8-a013-92964a4b71d6","Type":"ContainerDied","Data":"8a02a7daac6c4a7a648f47e7fa5203884e806e7b2f201de5acb089aa0085efd9"} Oct 30 22:16:16.554669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.554635 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:16.680022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.679943 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\") pod \"b3f571ad-c298-4ce8-a013-92964a4b71d6\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " Oct 30 22:16:16.680251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.680030 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\") pod \"b3f571ad-c298-4ce8-a013-92964a4b71d6\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " Oct 30 22:16:16.680251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.680126 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\" (UniqueName: \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\") pod \"b3f571ad-c298-4ce8-a013-92964a4b71d6\" (UID: \"b3f571ad-c298-4ce8-a013-92964a4b71d6\") " Oct 30 22:16:16.681042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.680990 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca" pod "b3f571ad-c298-4ce8-a013-92964a4b71d6" (UID: "b3f571ad-c298-4ce8-a013-92964a4b71d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:16.683056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.683022 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca" pod "b3f571ad-c298-4ce8-a013-92964a4b71d6" (UID: "b3f571ad-c298-4ce8-a013-92964a4b71d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:16.684711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.684684 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk" pod "b3f571ad-c298-4ce8-a013-92964a4b71d6" (UID: "b3f571ad-c298-4ce8-a013-92964a4b71d6"). InnerVolumeSpecName "kube-api-access-n65fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:16.781393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.781357 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65fk\" (UniqueName: \"kubernetes.io/projected/b3f571ad-c298-4ce8-a013-92964a4b71d6-kube-api-access-n65fk\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:16.781393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.781389 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:16.781610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:16.781403 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f571ad-c298-4ce8-a013-92964a4b71d6-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:17.407351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:17.407316 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" Oct 30 22:16:17.407540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:17.407351 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94" event={"ID":"b3f571ad-c298-4ce8-a013-92964a4b71d6","Type":"ContainerDied","Data":"c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f"} Oct 30 22:16:17.407540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:17.407408 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c166df37238f7e2b79b87c8c735448fea4416e3c9dafe100ca0b974aa8a0121f" Oct 30 22:16:19.102780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.102749 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:16:19.103705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.103682 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b3f571ad-c298-4ce8-a013-92964a4b71d6" containerName="revoke-keys" Oct 30 22:16:19.103705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.103704 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f571ad-c298-4ce8-a013-92964a4b71d6" containerName="revoke-keys" Oct 30 22:16:19.103864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.103842 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f571ad-c298-4ce8-a013-92964a4b71d6" containerName="revoke-keys" Oct 30 22:16:19.107539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.107515 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.153671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.153648 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:16:19.157547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.157532 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.204750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.204721 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:16:19.206269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206238 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206298 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206361 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206471 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206511 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206535 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206607 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206632 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206664 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206701 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206747 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hp8\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206776 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.206917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206824 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.206917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206881 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.207048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.206973 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.207048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.207030 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97mr\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.207161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.207069 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.207161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.207143 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.221595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.221558 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:16:19.308180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308148 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308197 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308231 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308256 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308271 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308301 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308366 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308414 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hp8\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308440 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308485 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308540 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308657 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308692 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308709 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97mr\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.308768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308719 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308781 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308862 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.308882 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309083 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309095 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309139 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309190 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309217 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309217 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.309735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309534 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.309735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309677 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.309954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.309933 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.310131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.310105 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.310785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.310766 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.310962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.310940 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.311336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.311313 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.311727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.311711 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.373970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.373912 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97mr\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.374126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.374108 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.374267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.374245 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.374317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.374302 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hp8\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\") pod \"e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.420716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.420691 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:19.471729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.471699 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:19.857131 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:19.857098 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce09c22_dcd7_4141_a38c_a747590c3e89.slice/crio-412f13acffbed2ec169b33dd6505147e97af947fd011a6040a0df0751b3d8e06 WatchSource:0}: Error finding container 412f13acffbed2ec169b33dd6505147e97af947fd011a6040a0df0751b3d8e06: Status 404 returned error can't find the container with id 412f13acffbed2ec169b33dd6505147e97af947fd011a6040a0df0751b3d8e06 Oct 30 22:16:19.858191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.858167 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:16:19.859650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.859621 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:19.859723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.859681 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:19.859765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.859722 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:19.915892 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:19.915865 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d0c3d0_48b2_44fc_a24f_f9df4da3583e.slice/crio-2b50d7e04fea4471158d533868b35c88275b6be23ec7ba66979ab879dcb60af4 WatchSource:0}: Error finding container 2b50d7e04fea4471158d533868b35c88275b6be23ec7ba66979ab879dcb60af4: Status 404 returned error can't find the container with id 2b50d7e04fea4471158d533868b35c88275b6be23ec7ba66979ab879dcb60af4 Oct 30 22:16:19.916079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.916058 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:16:19.917717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.917688 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:19.917794 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.917752 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:19.917794 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:19.917782 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:20.429222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.429183 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" event={"ID":"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e","Type":"ContainerStarted","Data":"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f"} Oct 30 22:16:20.430109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.429253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" event={"ID":"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e","Type":"ContainerStarted","Data":"2b50d7e04fea4471158d533868b35c88275b6be23ec7ba66979ab879dcb60af4"} Oct 30 22:16:20.430921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.430894 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" event={"ID":"dce09c22-dcd7-4141-a38c-a747590c3e89","Type":"ContainerStarted","Data":"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68"} Oct 30 22:16:20.431061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.430942 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" event={"ID":"dce09c22-dcd7-4141-a38c-a747590c3e89","Type":"ContainerStarted","Data":"412f13acffbed2ec169b33dd6505147e97af947fd011a6040a0df0751b3d8e06"} Oct 30 22:16:20.472901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.472871 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:20.474857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.474828 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.474997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.474872 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:20.557982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.557924 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podStartSLOduration=1.557908124 podStartE2EDuration="1.557908124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:20.554784243 +0000 UTC m=+1308.523603834" watchObservedRunningTime="2026-10-30 22:16:20.557908124 +0000 UTC m=+1308.526727689" Oct 30 22:16:20.730659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:20.730433 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podStartSLOduration=1.730412035 podStartE2EDuration="1.730412035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:20.702884497 +0000 UTC m=+1308.671704061" watchObservedRunningTime="2026-10-30 22:16:20.730412035 +0000 UTC m=+1308.699231601" Oct 30 22:16:21.429597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:21.428767 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:21.429597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:21.428954 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.429597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:21.429000 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:21.472546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:21.472507 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.472737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:21.472566 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:22.438440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:22.437875 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.438440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:22.437937 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:22.474850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:22.474064 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.474850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:22.474215 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:23.422599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:23.421940 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.422599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:23.422001 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:23.472954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:23.472922 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.473336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:23.472999 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:24.424598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:24.423965 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.424598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:24.424031 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:24.472216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:24.472186 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.472367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:24.472233 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:25.424598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:25.423953 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.424598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:25.424028 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:25.474457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:25.474417 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.474670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:25.474488 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:26.421560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:26.421515 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:16:26.421775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:26.421606 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:16:26.472586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:26.472544 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:16:26.473093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:26.472626 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:16:26.640618 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:16:26.640563 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:16:27.242217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.242119 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5"] Oct 30 22:16:27.247451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.247424 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.302434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.302390 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:27.356111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.356082 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:16:27.356387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.356368 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:27.356529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.356516 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:27.360688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.360471 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5"] Oct 30 22:16:27.361686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.361655 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbx6\" (UniqueName: \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.361806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.361789 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.361936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.361916 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.425373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.425346 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:27.425714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.425693 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:27.426373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.426355 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:16:27.463221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.463189 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.463388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.463281 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.463388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.463361 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbx6\" (UniqueName: \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.463901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.463874 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.465642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.465622 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.477257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.477235 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:27.477543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.477516 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:27.478280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.478260 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:16:27.526762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.526711 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbx6\" (UniqueName: \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\") pod \"maas-api-revoke-sub-e7466096-mpnq5\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.569647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.569625 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:27.979044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:27.978463 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5"] Oct 30 22:16:28.496938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:28.496906 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" event={"ID":"cf441fe2-cde1-4a3b-b372-038ae1cb9058","Type":"ContainerStarted","Data":"192d52f32ae838409595ff5b5228c5eda288b3e0772e53fe82e85ba9d771d79c"} Oct 30 22:16:28.497333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:28.496955 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" event={"ID":"cf441fe2-cde1-4a3b-b372-038ae1cb9058","Type":"ContainerStarted","Data":"d71c88fdae3e6b65cca3a56e0680600ef146eb3d64808af9afa77f1ebe7b4cfe"} Oct 30 22:16:28.630318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:28.630258 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" podStartSLOduration=1.630245968 podStartE2EDuration="1.630245968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:28.627170601 +0000 UTC m=+1316.595990165" watchObservedRunningTime="2026-10-30 22:16:28.630245968 +0000 UTC m=+1316.599065532" Oct 30 22:16:29.505214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:29.505174 4292 generic.go:356] "Generic (PLEG): container finished" podID="cf441fe2-cde1-4a3b-b372-038ae1cb9058" containerID="192d52f32ae838409595ff5b5228c5eda288b3e0772e53fe82e85ba9d771d79c" exitCode=0 Oct 30 22:16:29.505614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:29.505253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" event={"ID":"cf441fe2-cde1-4a3b-b372-038ae1cb9058","Type":"ContainerDied","Data":"192d52f32ae838409595ff5b5228c5eda288b3e0772e53fe82e85ba9d771d79c"} Oct 30 22:16:30.023033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.022995 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c"] Oct 30 22:16:30.027445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.027426 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.057220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.057189 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd"] Oct 30 22:16:30.065625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.065597 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8814f4b4-wpptd"] Oct 30 22:16:30.086884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.086858 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c"] Oct 30 22:16:30.148192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.148159 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530d73d6-51b8-40d5-b5a4-f104271bbefe" path="/var/lib/kubelet/pods/530d73d6-51b8-40d5-b5a4-f104271bbefe/volumes" Oct 30 22:16:30.204451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.204423 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.204613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.204498 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.204657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.204632 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmd5\" (UniqueName: \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.306441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.306412 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.306591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.306501 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.306679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.306599 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmd5\" (UniqueName: \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.307201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.307177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.308739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.308718 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.369735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.369711 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmd5\" (UniqueName: \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\") pod \"maas-api-revoke-sub-09e5677b-pth7c\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.641058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.641029 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:30.686883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.686853 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:30.813796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.813758 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\" (UniqueName: \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\") pod \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " Oct 30 22:16:30.813955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.813810 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\") pod \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " Oct 30 22:16:30.813955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.813855 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\") pod \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\" (UID: \"cf441fe2-cde1-4a3b-b372-038ae1cb9058\") " Oct 30 22:16:30.814651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.814619 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca" pod "cf441fe2-cde1-4a3b-b372-038ae1cb9058" (UID: "cf441fe2-cde1-4a3b-b372-038ae1cb9058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:30.816459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.816433 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6" pod "cf441fe2-cde1-4a3b-b372-038ae1cb9058" (UID: "cf441fe2-cde1-4a3b-b372-038ae1cb9058"). InnerVolumeSpecName "kube-api-access-spbx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:30.816794 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.816772 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca" pod "cf441fe2-cde1-4a3b-b372-038ae1cb9058" (UID: "cf441fe2-cde1-4a3b-b372-038ae1cb9058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:30.915269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.915191 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbx6\" (UniqueName: \"kubernetes.io/projected/cf441fe2-cde1-4a3b-b372-038ae1cb9058-kube-api-access-spbx6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.915269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.915224 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.915269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:30.915238 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf441fe2-cde1-4a3b-b372-038ae1cb9058-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.025904 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:31.025866 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ab0f80_899c_4bf7_b001_3d17ce7d3ff6.slice/crio-2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850 WatchSource:0}: Error finding container 2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850: Status 404 returned error can't find the container with id 2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850 Oct 30 22:16:31.026301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.026276 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c"] Oct 30 22:16:31.520934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.520846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" event={"ID":"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6","Type":"ContainerStarted","Data":"6830d57de87b3e334c40fc28abbc3fc3297f62a55e7777f7f6307424ce9219e6"} Oct 30 22:16:31.520934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.520901 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" event={"ID":"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6","Type":"ContainerStarted","Data":"2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850"} Oct 30 22:16:31.522357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.522330 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" Oct 30 22:16:31.522475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.522368 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5" event={"ID":"cf441fe2-cde1-4a3b-b372-038ae1cb9058","Type":"ContainerDied","Data":"d71c88fdae3e6b65cca3a56e0680600ef146eb3d64808af9afa77f1ebe7b4cfe"} Oct 30 22:16:31.522475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.522410 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71c88fdae3e6b65cca3a56e0680600ef146eb3d64808af9afa77f1ebe7b4cfe" Oct 30 22:16:31.645177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:31.645122 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" podStartSLOduration=2.645108984 podStartE2EDuration="2.645108984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:31.642188141 +0000 UTC m=+1319.611007705" watchObservedRunningTime="2026-10-30 22:16:31.645108984 +0000 UTC m=+1319.613928584" Oct 30 22:16:32.564600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:32.554922 4292 generic.go:356] "Generic (PLEG): container finished" podID="60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" containerID="6830d57de87b3e334c40fc28abbc3fc3297f62a55e7777f7f6307424ce9219e6" exitCode=0 Oct 30 22:16:32.564600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:32.554999 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" event={"ID":"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6","Type":"ContainerDied","Data":"6830d57de87b3e334c40fc28abbc3fc3297f62a55e7777f7f6307424ce9219e6"} Oct 30 22:16:33.567598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:33.566913 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:33.674598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:33.673655 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:33.871754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:33.870673 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.152686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.143394 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 22:16:34.185634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.176617 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.185634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.176801 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.212658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.197057 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.252191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.248642 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:16:34.467408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.441480 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.475173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.473932 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.626600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.622647 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.645776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.645747 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:34.645958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.645934 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:16:34.708041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.674925 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:34.708041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.677138 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:16:34.847603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.847141 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\") pod \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " Oct 30 22:16:34.847603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.847240 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\") pod \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " Oct 30 22:16:34.847603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.847441 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\" (UniqueName: \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\") pod \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\" (UID: \"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6\") " Oct 30 22:16:34.870523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.870026 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca" pod "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" (UID: "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:34.896639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.879765 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.896639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.881085 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:16:34.896639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.881377 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:16:34.903834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.901271 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" event={"ID":"60ab0f80-899c-4bf7-b001-3d17ce7d3ff6","Type":"ContainerDied","Data":"2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850"} Oct 30 22:16:34.903834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.901333 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2354eb9b4b454a75535c3fc360b8088fe0fc160e9dce858449f4262efdaab850" Oct 30 22:16:34.965600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.951185 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5" pod "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" (UID: "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6"). InnerVolumeSpecName "kube-api-access-zdmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.965600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:34.960916 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:16:35.022456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.018284 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmd5\" (UniqueName: \"kubernetes.io/projected/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-kube-api-access-zdmd5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.132866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.132786 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca" pod "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" (UID: "60ab0f80-899c-4bf7-b001-3d17ce7d3ff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:35.164175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.164137 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 22:16:35.164839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.164697 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:35.165751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.165733 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:16:35.182605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.181895 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:16:35.225594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.221647 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:16:35.225594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.222406 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:16:35.257595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.255667 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.301589 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.301539 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:16:35.309872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.309852 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:16:35.338448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.338413 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:16:35.908999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:35.908968 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c" Oct 30 22:16:41.058899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:41.058868 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv"] Oct 30 22:16:41.108337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:41.108310 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c66426-hflpv"] Oct 30 22:16:42.146922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.146892 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d704afd-e3e9-4f8d-817a-673b6d2c6f2c" path="/var/lib/kubelet/pods/0d704afd-e3e9-4f8d-817a-673b6d2c6f2c/volumes" Oct 30 22:16:42.615193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.615142 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x"] Oct 30 22:16:42.616238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616212 4292 cpu_manager.go:414] "containerMap: removing container" podUID="60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" containerName="revoke-keys" Oct 30 22:16:42.616238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616237 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" containerName="revoke-keys" Oct 30 22:16:42.616441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616273 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cf441fe2-cde1-4a3b-b372-038ae1cb9058" containerName="revoke-keys" Oct 30 22:16:42.616441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616282 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf441fe2-cde1-4a3b-b372-038ae1cb9058" containerName="revoke-keys" Oct 30 22:16:42.616606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616562 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" containerName="revoke-keys" Oct 30 22:16:42.616606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.616599 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf441fe2-cde1-4a3b-b372-038ae1cb9058" containerName="revoke-keys" Oct 30 22:16:42.623499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.623475 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.645892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.645867 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.646025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.645904 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.646095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.646065 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqd5\" (UniqueName: \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.676783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.676757 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x"] Oct 30 22:16:42.722134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.722109 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:42.722134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.722110 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:42.722134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.722110 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:42.727478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.727457 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:16:42.747451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.747428 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqd5\" (UniqueName: \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.747722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.747693 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.747831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.747733 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.748476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.748434 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.750326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.750303 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.810342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.810318 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqd5\" (UniqueName: \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\") pod \"maas-api-revoke-sub-633c6a71-x6f2x\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:42.937705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:42.937636 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:43.319189 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:43.319155 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7edbe1_5f79_420a_923f_ee32ca22f1c4.slice/crio-0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48 WatchSource:0}: Error finding container 0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48: Status 404 returned error can't find the container with id 0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48 Oct 30 22:16:43.319634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:43.319368 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x"] Oct 30 22:16:43.967864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:43.967774 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" event={"ID":"9c7edbe1-5f79-420a-923f-ee32ca22f1c4","Type":"ContainerStarted","Data":"30de657658caaf4268527fc58c626a81c92f073d1dcf5a8339529523ea956115"} Oct 30 22:16:43.967864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:43.967838 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" event={"ID":"9c7edbe1-5f79-420a-923f-ee32ca22f1c4","Type":"ContainerStarted","Data":"0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48"} Oct 30 22:16:44.976980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:44.976947 4292 generic.go:356] "Generic (PLEG): container finished" podID="9c7edbe1-5f79-420a-923f-ee32ca22f1c4" containerID="30de657658caaf4268527fc58c626a81c92f073d1dcf5a8339529523ea956115" exitCode=0 Oct 30 22:16:44.977362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:44.977029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" event={"ID":"9c7edbe1-5f79-420a-923f-ee32ca22f1c4","Type":"ContainerDied","Data":"30de657658caaf4268527fc58c626a81c92f073d1dcf5a8339529523ea956115"} Oct 30 22:16:45.234764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.234690 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:16:45.239291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.239266 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.278737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.278706 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.278876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.278836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.278936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.278915 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.278996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.278954 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.279065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.279045 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.279127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.279087 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.279184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.279172 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.279240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.279216 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fln\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.279313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.279298 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.335320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.335290 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:16:45.381074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381035 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381092 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fln\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381130 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381213 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381323 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381386 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381413 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381477 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381625 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381701 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381834 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381565 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.381909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.381894 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.383715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.383697 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.383807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.383792 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.444547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.444524 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.445963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.445941 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fln\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\") pod \"e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.552843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.552820 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:45.646295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.646261 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27"] Oct 30 22:16:45.651555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.651533 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.685974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.685944 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.686143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.686121 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.686248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.686233 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtv4z\" (UniqueName: \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.708446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.708419 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27"] Oct 30 22:16:45.787380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.787347 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.787557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.787419 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtv4z\" (UniqueName: \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.787557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.787505 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.788170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.788142 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.789674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.789654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.855682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.855624 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtv4z\" (UniqueName: \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\") pod \"maas-api-revoke-sub-4e666106-smm27\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.970347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.970315 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:45.988465 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:45.988439 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23078e7c_36a3_4bb0_a582_b72f4faad4b0.slice/crio-ec96693e0bab4b47952e4deb6c32e277572be68655238992f3e66272d61d9d4e WatchSource:0}: Error finding container ec96693e0bab4b47952e4deb6c32e277572be68655238992f3e66272d61d9d4e: Status 404 returned error can't find the container with id ec96693e0bab4b47952e4deb6c32e277572be68655238992f3e66272d61d9d4e Oct 30 22:16:45.988926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.988903 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:16:45.991162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.991118 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:45.991263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.991206 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:45.991263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:45.991252 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:46.160374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.160349 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:46.294993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.294941 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\") pod \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " Oct 30 22:16:46.295185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.295022 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\" (UniqueName: \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\") pod \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " Oct 30 22:16:46.295249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.295236 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\") pod \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\" (UID: \"9c7edbe1-5f79-420a-923f-ee32ca22f1c4\") " Oct 30 22:16:46.295792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.295741 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca" pod "9c7edbe1-5f79-420a-923f-ee32ca22f1c4" (UID: "9c7edbe1-5f79-420a-923f-ee32ca22f1c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:46.297528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.297500 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca" pod "9c7edbe1-5f79-420a-923f-ee32ca22f1c4" (UID: "9c7edbe1-5f79-420a-923f-ee32ca22f1c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:46.298297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.298272 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5" pod "9c7edbe1-5f79-420a-923f-ee32ca22f1c4" (UID: "9c7edbe1-5f79-420a-923f-ee32ca22f1c4"). InnerVolumeSpecName "kube-api-access-bvqd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:46.392726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.392691 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27"] Oct 30 22:16:46.393045 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:46.393020 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf58b6d_d909_453d_985b_e81c6bada8cc.slice/crio-561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3 WatchSource:0}: Error finding container 561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3: Status 404 returned error can't find the container with id 561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3 Oct 30 22:16:46.396706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.396687 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvqd5\" (UniqueName: \"kubernetes.io/projected/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-kube-api-access-bvqd5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.396768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.396709 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.396768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:46.396721 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7edbe1-5f79-420a-923f-ee32ca22f1c4-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.021671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.021638 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" Oct 30 22:16:47.022090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.021635 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x" event={"ID":"9c7edbe1-5f79-420a-923f-ee32ca22f1c4","Type":"ContainerDied","Data":"0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48"} Oct 30 22:16:47.022090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.021801 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0207e9f92e3b6bdab5bb140b0bf99f25c05406533058115f3c4645658ddaca48" Oct 30 22:16:47.023519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.023460 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" event={"ID":"ecf58b6d-d909-453d-985b-e81c6bada8cc","Type":"ContainerStarted","Data":"288eb44504a0f218180b3f322db6e6d78f6c11c79308c24bf9fd20989c73ea48"} Oct 30 22:16:47.023519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.023511 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" event={"ID":"ecf58b6d-d909-453d-985b-e81c6bada8cc","Type":"ContainerStarted","Data":"561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3"} Oct 30 22:16:47.025501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.025475 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" event={"ID":"23078e7c-36a3-4bb0-a582-b72f4faad4b0","Type":"ContainerStarted","Data":"ca4355df6310863ded04367f99790fa24a6a5ea4dfb906baaf5c203ca4726fb0"} Oct 30 22:16:47.025644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.025524 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" event={"ID":"23078e7c-36a3-4bb0-a582-b72f4faad4b0","Type":"ContainerStarted","Data":"ec96693e0bab4b47952e4deb6c32e277572be68655238992f3e66272d61d9d4e"} Oct 30 22:16:47.157290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.157181 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" podStartSLOduration=2.157167227 podStartE2EDuration="2.157167227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:47.155735707 +0000 UTC m=+1335.124555270" watchObservedRunningTime="2026-10-30 22:16:47.157167227 +0000 UTC m=+1335.125986823" Oct 30 22:16:47.339470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.339401 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podStartSLOduration=2.339382843 podStartE2EDuration="2.339382843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:47.337314443 +0000 UTC m=+1335.306134008" watchObservedRunningTime="2026-10-30 22:16:47.339382843 +0000 UTC m=+1335.308202410" Oct 30 22:16:47.553826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.553796 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:47.555689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.555665 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:47.555814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:47.555710 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:48.034801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.034719 4292 generic.go:356] "Generic (PLEG): container finished" podID="ecf58b6d-d909-453d-985b-e81c6bada8cc" containerID="288eb44504a0f218180b3f322db6e6d78f6c11c79308c24bf9fd20989c73ea48" exitCode=0 Oct 30 22:16:48.035275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.034811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" event={"ID":"ecf58b6d-d909-453d-985b-e81c6bada8cc","Type":"ContainerDied","Data":"288eb44504a0f218180b3f322db6e6d78f6c11c79308c24bf9fd20989c73ea48"} Oct 30 22:16:48.553919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.553883 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:48.554097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.553936 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:48.771488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.771452 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:16:48.772090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.772076 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9c7edbe1-5f79-420a-923f-ee32ca22f1c4" containerName="revoke-keys" Oct 30 22:16:48.772184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.772092 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7edbe1-5f79-420a-923f-ee32ca22f1c4" containerName="revoke-keys" Oct 30 22:16:48.772236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.772228 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7edbe1-5f79-420a-923f-ee32ca22f1c4" containerName="revoke-keys" Oct 30 22:16:48.775797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.775780 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840237 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840400 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840433 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840457 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840516 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840648 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840697 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840761 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.840810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.840789 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cpq\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.909597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.898828 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:16:48.941755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.941722 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.941949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.941760 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cpq\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.941949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.941830 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.941949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.941939 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.941972 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942002 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942024 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942079 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942131 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942175 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942470 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942546 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.942910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.942890 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.943193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.943166 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.944792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.944768 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:48.945174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:48.945154 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:49.011972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.011933 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:49.012878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.012850 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cpq\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\") pod \"e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:49.113194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.112968 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:49.228365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.227056 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:49.347551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.346986 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\") pod \"ecf58b6d-d909-453d-985b-e81c6bada8cc\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " Oct 30 22:16:49.347551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.347183 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\" (UniqueName: \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\") pod \"ecf58b6d-d909-453d-985b-e81c6bada8cc\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " Oct 30 22:16:49.348170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.348136 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\") pod \"ecf58b6d-d909-453d-985b-e81c6bada8cc\" (UID: \"ecf58b6d-d909-453d-985b-e81c6bada8cc\") " Oct 30 22:16:49.350308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.350280 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca" pod "ecf58b6d-d909-453d-985b-e81c6bada8cc" (UID: "ecf58b6d-d909-453d-985b-e81c6bada8cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:49.352494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.352466 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca" pod "ecf58b6d-d909-453d-985b-e81c6bada8cc" (UID: "ecf58b6d-d909-453d-985b-e81c6bada8cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:49.352625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.352569 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z" pod "ecf58b6d-d909-453d-985b-e81c6bada8cc" (UID: "ecf58b6d-d909-453d-985b-e81c6bada8cc"). InnerVolumeSpecName "kube-api-access-jtv4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:49.450063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.449990 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtv4z\" (UniqueName: \"kubernetes.io/projected/ecf58b6d-d909-453d-985b-e81c6bada8cc-kube-api-access-jtv4z\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:49.450063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.450016 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:49.450063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.450028 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf58b6d-d909-453d-985b-e81c6bada8cc-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:49.553646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.553610 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:49.553838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.553682 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:49.614721 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:16:49.614683 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96bd55b_3272_406e_bb54_11d797566ef4.slice/crio-ce2083fbd90ca04a7b8c3b2e6980ad1a2348e4e821ceca1d5d5e0e646a11ca80 WatchSource:0}: Error finding container ce2083fbd90ca04a7b8c3b2e6980ad1a2348e4e821ceca1d5d5e0e646a11ca80: Status 404 returned error can't find the container with id ce2083fbd90ca04a7b8c3b2e6980ad1a2348e4e821ceca1d5d5e0e646a11ca80 Oct 30 22:16:49.616310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.616277 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:16:49.617314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.617285 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:49.617369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.617348 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:49.617408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:49.617378 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:50.051870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.051836 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" Oct 30 22:16:50.051870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.051856 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27" event={"ID":"ecf58b6d-d909-453d-985b-e81c6bada8cc","Type":"ContainerDied","Data":"561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3"} Oct 30 22:16:50.052107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.051914 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561bf35103f3621e69d648ca46b036696945ef1d3057f98438ff89c067b5a5c3" Oct 30 22:16:50.053849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.053814 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" event={"ID":"b96bd55b-3272-406e-bb54-11d797566ef4","Type":"ContainerStarted","Data":"86a81dd70017384861bbd2411ffe8341f4d4ec9ae2275df3e2a843d4e7793024"} Oct 30 22:16:50.053966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.053864 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" event={"ID":"b96bd55b-3272-406e-bb54-11d797566ef4","Type":"ContainerStarted","Data":"ce2083fbd90ca04a7b8c3b2e6980ad1a2348e4e821ceca1d5d5e0e646a11ca80"} Oct 30 22:16:50.114102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.114029 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:50.116124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.116093 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:50.116277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.116169 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:50.179039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.178990 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podStartSLOduration=2.178978401 podStartE2EDuration="2.178978401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.178762348 +0000 UTC m=+1338.147581912" watchObservedRunningTime="2026-10-30 22:16:50.178978401 +0000 UTC m=+1338.147797965" Oct 30 22:16:50.553552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.553516 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:50.553755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:50.553603 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:51.118599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:51.117982 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.118599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:51.118045 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:51.553280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:51.553247 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.553438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:51.553302 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:52.115731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:52.115693 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.115932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:52.115762 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:52.553586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:52.553538 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.554022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:52.553609 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:16:53.116792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:53.116740 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.116975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:53.116805 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:53.557497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:53.557467 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:53.557960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:53.557938 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:53.558548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:53.558527 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:16:54.114284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:54.114246 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.114451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:54.114312 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:55.061810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:55.061778 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm"] Oct 30 22:16:55.122740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:55.122699 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.122998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:55.122969 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:55.134243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:55.134209 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2cfac8-9z5vm"] Oct 30 22:16:56.123599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.122963 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:56.123599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.123030 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:56.208033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.207992 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135c6e87-cb11-4fce-9901-26f5a01fc5f4" path="/var/lib/kubelet/pods/135c6e87-cb11-4fce-9901-26f5a01fc5f4/volumes" Oct 30 22:16:56.493657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.491643 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw"] Oct 30 22:16:56.493657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.492594 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ecf58b6d-d909-453d-985b-e81c6bada8cc" containerName="revoke-keys" Oct 30 22:16:56.493657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.492616 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf58b6d-d909-453d-985b-e81c6bada8cc" containerName="revoke-keys" Oct 30 22:16:56.493657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.492788 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf58b6d-d909-453d-985b-e81c6bada8cc" containerName="revoke-keys" Oct 30 22:16:56.498329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.498306 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:56.526699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.526670 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw"] Oct 30 22:16:56.579081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.579050 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:56.579397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.579375 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:16:56.579628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.579610 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:56.600606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.590441 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:56.605788 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.605755 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdcc\" (UniqueName: \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:56.765653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.737020 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:56.766057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.655964 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:56.978597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.978449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.001613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.988676 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.001613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.988790 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdcc\" (UniqueName: \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.001613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:56.989118 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.027650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.026343 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.033760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.033723 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.080492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.080456 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdcc\" (UniqueName: \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\") pod \"maas-api-revoke-sub-3d5ecf97-9dpdw\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.093066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.093021 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.123440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.123278 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:57.123440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.123339 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:16:57.174845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.174809 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:16:57.188687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.187831 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.320844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.268160 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.356600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.330664 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.409605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.384369 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.410282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.392347 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:16:57.442821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.392391 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.451318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.393653 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:16:57.465417 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:16:57.442221 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:16:57.734650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.726963 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.762002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.736493 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:57.985648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:57.979061 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:58.195162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.195120 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 22:16:58.236070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.233481 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:16:58.484159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.415967 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:16:58.484159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.449472 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:58.484159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.467336 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:16:58.493442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.493373 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:16:58.493647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.493622 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:58.500386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.500371 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:16:58.500590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.500562 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:16:58.500742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.500730 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:16:58.500878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.500866 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:16:58.617830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.617794 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:16:58.618091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.618059 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:58.775657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.775423 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:16:58.981105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.981068 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:16:58.997325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:58.997294 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:59.099991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.099967 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:16:59.102823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.102805 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:59.138292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.138267 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:16:59.148567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.148542 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j"] Oct 30 22:16:59.156441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.156414 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.160495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.160476 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j"] Oct 30 22:16:59.170384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.170362 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw"] Oct 30 22:16:59.174783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.174756 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4"] Oct 30 22:16:59.180058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.180036 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23cd8b01-tqtn4"] Oct 30 22:16:59.358854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.358822 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh55r\" (UniqueName: \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.359211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.358869 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.359211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.359099 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.460931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.460898 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.461074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.460975 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh55r\" (UniqueName: \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.461074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.461021 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.461684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.461659 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.463236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.463214 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.524381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.524347 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh55r\" (UniqueName: \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\") pod \"maas-api-revoke-sub-b0e23e2f-c9r4j\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.820370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.820344 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:16:59.916686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.916647 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" event={"ID":"3bd0b5a0-6be5-4675-ba87-a836e7e15966","Type":"ContainerStarted","Data":"5002f8008d1da5b909fe078684fe12a79aa3092ed55aae57ecc94c8b555e8e27"} Oct 30 22:16:59.916869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:16:59.916710 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" event={"ID":"3bd0b5a0-6be5-4675-ba87-a836e7e15966","Type":"ContainerStarted","Data":"36104e8733b3b6bd853568c5786b26dacad84fc9fd92bfea798367f25ca59d17"} Oct 30 22:17:00.039867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.039814 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" podStartSLOduration=4.039799933 podStartE2EDuration="4.039799933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:00.036941117 +0000 UTC m=+1348.005760681" watchObservedRunningTime="2026-10-30 22:17:00.039799933 +0000 UTC m=+1348.008619496" Oct 30 22:17:00.148001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.147976 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4a4333-3032-4de8-a516-fca6b5ea9719" path="/var/lib/kubelet/pods/fb4a4333-3032-4de8-a516-fca6b5ea9719/volumes" Oct 30 22:17:00.201628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.201595 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j"] Oct 30 22:17:00.233509 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:00.233480 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1b1ba_2a4c_4eb3_a8cc_0254041f03e8.slice/crio-c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b WatchSource:0}: Error finding container c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b: Status 404 returned error can't find the container with id c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b Oct 30 22:17:00.665545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.665504 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt"] Oct 30 22:17:00.670175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.670146 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.681025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.679524 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.681025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.679656 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.681025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.679711 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7zz\" (UniqueName: \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.731789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.731757 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt"] Oct 30 22:17:00.781438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.781358 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.781608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.781449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.781608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.781472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7zz\" (UniqueName: \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.782187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.782159 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.783841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.783822 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.849657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.849635 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7zz\" (UniqueName: \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\") pod \"maas-api-revoke-sub-53093ac1-nkrrt\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:00.925850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.925822 4292 generic.go:356] "Generic (PLEG): container finished" podID="3bd0b5a0-6be5-4675-ba87-a836e7e15966" containerID="5002f8008d1da5b909fe078684fe12a79aa3092ed55aae57ecc94c8b555e8e27" exitCode=0 Oct 30 22:17:00.925985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.925909 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" event={"ID":"3bd0b5a0-6be5-4675-ba87-a836e7e15966","Type":"ContainerDied","Data":"5002f8008d1da5b909fe078684fe12a79aa3092ed55aae57ecc94c8b555e8e27"} Oct 30 22:17:00.927724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.927699 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" event={"ID":"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8","Type":"ContainerStarted","Data":"2d46a72383804420ffacbd40ee0f70900ef591d85d4f063126bc22d721ffe753"} Oct 30 22:17:00.927842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.927738 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" event={"ID":"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8","Type":"ContainerStarted","Data":"c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b"} Oct 30 22:17:00.990195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:00.990169 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:01.166219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.166165 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" podStartSLOduration=3.166149336 podStartE2EDuration="3.166149336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:01.165324201 +0000 UTC m=+1349.134143768" watchObservedRunningTime="2026-10-30 22:17:01.166149336 +0000 UTC m=+1349.134968927" Oct 30 22:17:01.380049 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:01.380017 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426c23ee_2a52_4b38_9c92_817115d5674a.slice/crio-31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789 WatchSource:0}: Error finding container 31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789: Status 404 returned error can't find the container with id 31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789 Oct 30 22:17:01.381365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.381338 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt"] Oct 30 22:17:01.937382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.937343 4292 generic.go:356] "Generic (PLEG): container finished" podID="91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" containerID="2d46a72383804420ffacbd40ee0f70900ef591d85d4f063126bc22d721ffe753" exitCode=0 Oct 30 22:17:01.937829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.937429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" event={"ID":"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8","Type":"ContainerDied","Data":"2d46a72383804420ffacbd40ee0f70900ef591d85d4f063126bc22d721ffe753"} Oct 30 22:17:01.939088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.939059 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" event={"ID":"426c23ee-2a52-4b38-9c92-817115d5674a","Type":"ContainerStarted","Data":"948016b98fce10e2c086afaab9ff71eb0b3201a03e282130490f5c9e8e8d2c05"} Oct 30 22:17:01.939180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:01.939113 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" event={"ID":"426c23ee-2a52-4b38-9c92-817115d5674a","Type":"ContainerStarted","Data":"31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789"} Oct 30 22:17:02.109139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.109109 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:17:02.190700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.190635 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" podStartSLOduration=2.190616759 podStartE2EDuration="2.190616759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:02.187637878 +0000 UTC m=+1350.156457441" watchObservedRunningTime="2026-10-30 22:17:02.190616759 +0000 UTC m=+1350.159436322" Oct 30 22:17:02.202638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.202606 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\") pod \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " Oct 30 22:17:02.202839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.202673 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\" (UniqueName: \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\") pod \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " Oct 30 22:17:02.202912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.202838 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\") pod \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\" (UID: \"3bd0b5a0-6be5-4675-ba87-a836e7e15966\") " Oct 30 22:17:02.203158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.203123 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca" pod "3bd0b5a0-6be5-4675-ba87-a836e7e15966" (UID: "3bd0b5a0-6be5-4675-ba87-a836e7e15966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:02.203350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.203334 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:02.204784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.204762 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca" pod "3bd0b5a0-6be5-4675-ba87-a836e7e15966" (UID: "3bd0b5a0-6be5-4675-ba87-a836e7e15966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:02.204874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.204839 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc" pod "3bd0b5a0-6be5-4675-ba87-a836e7e15966" (UID: "3bd0b5a0-6be5-4675-ba87-a836e7e15966"). InnerVolumeSpecName "kube-api-access-txdcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:02.304178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.304146 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0b5a0-6be5-4675-ba87-a836e7e15966-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:02.304178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.304174 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txdcc\" (UniqueName: \"kubernetes.io/projected/3bd0b5a0-6be5-4675-ba87-a836e7e15966-kube-api-access-txdcc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:02.889318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.889281 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t"] Oct 30 22:17:02.890018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.889994 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3bd0b5a0-6be5-4675-ba87-a836e7e15966" containerName="revoke-keys" Oct 30 22:17:02.890018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.890017 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd0b5a0-6be5-4675-ba87-a836e7e15966" containerName="revoke-keys" Oct 30 22:17:02.890190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.890182 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd0b5a0-6be5-4675-ba87-a836e7e15966" containerName="revoke-keys" Oct 30 22:17:02.893710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.893690 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:02.912767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.912742 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:02.912850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.912793 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:02.912895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.912875 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fsft\" (UniqueName: \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:02.947372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.947337 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" Oct 30 22:17:02.947372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.947356 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw" event={"ID":"3bd0b5a0-6be5-4675-ba87-a836e7e15966","Type":"ContainerDied","Data":"36104e8733b3b6bd853568c5786b26dacad84fc9fd92bfea798367f25ca59d17"} Oct 30 22:17:02.948008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.947404 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36104e8733b3b6bd853568c5786b26dacad84fc9fd92bfea798367f25ca59d17" Oct 30 22:17:02.953700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.953676 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t"] Oct 30 22:17:02.958537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.958508 4292 generic.go:356] "Generic (PLEG): container finished" podID="426c23ee-2a52-4b38-9c92-817115d5674a" containerID="948016b98fce10e2c086afaab9ff71eb0b3201a03e282130490f5c9e8e8d2c05" exitCode=0 Oct 30 22:17:02.958707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:02.958651 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" event={"ID":"426c23ee-2a52-4b38-9c92-817115d5674a","Type":"ContainerDied","Data":"948016b98fce10e2c086afaab9ff71eb0b3201a03e282130490f5c9e8e8d2c05"} Oct 30 22:17:03.014409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.014354 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.014859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.014482 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.015702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.015025 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsft\" (UniqueName: \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.015702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.015391 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.017126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.017102 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.087047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.087013 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsft\" (UniqueName: \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\") pod \"maas-api-revoke-sub-0844ac23-4vk4t\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.123185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.123160 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:17:03.207365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.207301 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:03.217444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.217422 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\") pod \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " Oct 30 22:17:03.217627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.217608 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\") pod \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " Oct 30 22:17:03.217703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.217663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\" (UniqueName: \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\") pod \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\" (UID: \"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8\") " Oct 30 22:17:03.217867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.217842 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca" pod "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" (UID: "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:03.218254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.218229 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.219972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.219934 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca" pod "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" (UID: "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:03.220412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.220388 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r" pod "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" (UID: "91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8"). InnerVolumeSpecName "kube-api-access-lh55r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:03.320169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.320141 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.320169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.320171 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh55r\" (UniqueName: \"kubernetes.io/projected/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8-kube-api-access-lh55r\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.594259 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:03.594226 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc24a706_f079_4dce_97e5_95e7822ed722.slice/crio-56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2 WatchSource:0}: Error finding container 56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2: Status 404 returned error can't find the container with id 56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2 Oct 30 22:17:03.595224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.595199 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t"] Oct 30 22:17:03.967398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.967360 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" event={"ID":"cc24a706-f079-4dce-97e5-95e7822ed722","Type":"ContainerStarted","Data":"311729011fee6e667676d7e883f3ceebfd837889ea04776cbe7fc58c017697fb"} Oct 30 22:17:03.967880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.967431 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" event={"ID":"cc24a706-f079-4dce-97e5-95e7822ed722","Type":"ContainerStarted","Data":"56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2"} Oct 30 22:17:03.968854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.968832 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" Oct 30 22:17:03.968939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.968867 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j" event={"ID":"91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8","Type":"ContainerDied","Data":"c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b"} Oct 30 22:17:03.968939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:03.968911 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11d3e4287f6c5de82747f7577cee7853f414135bb541bcd0f188fad2396095b" Oct 30 22:17:04.098005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.097955 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" podStartSLOduration=2.097941536 podStartE2EDuration="2.097941536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:04.094431445 +0000 UTC m=+1352.063251010" watchObservedRunningTime="2026-10-30 22:17:04.097941536 +0000 UTC m=+1352.066761099" Oct 30 22:17:04.140849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.140824 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:04.230189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.230125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\") pod \"426c23ee-2a52-4b38-9c92-817115d5674a\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " Oct 30 22:17:04.230317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.230231 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\" (UniqueName: \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\") pod \"426c23ee-2a52-4b38-9c92-817115d5674a\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " Oct 30 22:17:04.230317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.230301 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\") pod \"426c23ee-2a52-4b38-9c92-817115d5674a\" (UID: \"426c23ee-2a52-4b38-9c92-817115d5674a\") " Oct 30 22:17:04.230670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.230644 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca" pod "426c23ee-2a52-4b38-9c92-817115d5674a" (UID: "426c23ee-2a52-4b38-9c92-817115d5674a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:04.232188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.232157 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca" pod "426c23ee-2a52-4b38-9c92-817115d5674a" (UID: "426c23ee-2a52-4b38-9c92-817115d5674a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:04.232317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.232293 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz" pod "426c23ee-2a52-4b38-9c92-817115d5674a" (UID: "426c23ee-2a52-4b38-9c92-817115d5674a"). InnerVolumeSpecName "kube-api-access-cn7zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:04.331680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.331640 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn7zz\" (UniqueName: \"kubernetes.io/projected/426c23ee-2a52-4b38-9c92-817115d5674a-kube-api-access-cn7zz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.331680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.331670 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.331872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.331685 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426c23ee-2a52-4b38-9c92-817115d5674a-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.978121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.978050 4292 generic.go:356] "Generic (PLEG): container finished" podID="cc24a706-f079-4dce-97e5-95e7822ed722" containerID="311729011fee6e667676d7e883f3ceebfd837889ea04776cbe7fc58c017697fb" exitCode=0 Oct 30 22:17:04.978706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.978133 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" event={"ID":"cc24a706-f079-4dce-97e5-95e7822ed722","Type":"ContainerDied","Data":"311729011fee6e667676d7e883f3ceebfd837889ea04776cbe7fc58c017697fb"} Oct 30 22:17:04.979660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.979637 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" Oct 30 22:17:04.979766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.979662 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt" event={"ID":"426c23ee-2a52-4b38-9c92-817115d5674a","Type":"ContainerDied","Data":"31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789"} Oct 30 22:17:04.979766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:04.979701 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c17040c1fa90f1a95843b6061be00d2a0b3f6df785e12d794aa995ac1a4789" Oct 30 22:17:05.161192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161162 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:17:05.161836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161818 4292 cpu_manager.go:414] "containerMap: removing container" podUID="91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" containerName="revoke-keys" Oct 30 22:17:05.161836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161835 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" containerName="revoke-keys" Oct 30 22:17:05.161969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161880 4292 cpu_manager.go:414] "containerMap: removing container" podUID="426c23ee-2a52-4b38-9c92-817115d5674a" containerName="revoke-keys" Oct 30 22:17:05.161969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161887 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426c23ee-2a52-4b38-9c92-817115d5674a" containerName="revoke-keys" Oct 30 22:17:05.162040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.161985 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" containerName="revoke-keys" Oct 30 22:17:05.162040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.162002 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="426c23ee-2a52-4b38-9c92-817115d5674a" containerName="revoke-keys" Oct 30 22:17:05.165301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.165279 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.231708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.229920 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:17:05.243023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.242997 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243076 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243179 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243202 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243229 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243378 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243438 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmznp\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.243595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.243567 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345185 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345260 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345284 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345324 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345368 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345408 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345479 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmznp\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345560 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345656 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345786 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.345965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.345849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.346067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.346004 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.346130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.346115 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.346606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.346568 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.348205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.348177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.348394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.348376 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.408077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.408036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.408910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.408892 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmznp\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.478559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.478532 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:05.813983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.813948 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks"] Oct 30 22:17:05.824830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.824796 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:05.875608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.875537 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks"] Oct 30 22:17:05.911141 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:05.911097 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9721985_ff96_4206_ab76_70cad8b4d602.slice/crio-c376a42d628c00c1cc1df123dff7b183a0826dc81a097d6d0aac93d96f41a60f WatchSource:0}: Error finding container c376a42d628c00c1cc1df123dff7b183a0826dc81a097d6d0aac93d96f41a60f: Status 404 returned error can't find the container with id c376a42d628c00c1cc1df123dff7b183a0826dc81a097d6d0aac93d96f41a60f Oct 30 22:17:05.913398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.913372 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:17:05.914035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.914003 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:05.914139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.914064 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:05.914139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.914089 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:05.955407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.955363 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:05.955551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.955529 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2xz\" (UniqueName: \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:05.955655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:05.955622 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.047369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.047331 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" event={"ID":"c9721985-ff96-4206-ab76-70cad8b4d602","Type":"ContainerStarted","Data":"c376a42d628c00c1cc1df123dff7b183a0826dc81a097d6d0aac93d96f41a60f"} Oct 30 22:17:06.057349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.057321 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.057639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.057622 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2xz\" (UniqueName: \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.057788 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.057772 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.060434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.060409 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.063933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.063893 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.122833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.122750 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2xz\" (UniqueName: \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\") pod \"maas-api-revoke-sub-47b971b2-74wks\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.151540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.145664 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:06.241272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.241242 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:06.266737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.266703 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\") pod \"cc24a706-f079-4dce-97e5-95e7822ed722\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " Oct 30 22:17:06.266885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.266790 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\" (UniqueName: \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\") pod \"cc24a706-f079-4dce-97e5-95e7822ed722\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " Oct 30 22:17:06.266885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.266840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\") pod \"cc24a706-f079-4dce-97e5-95e7822ed722\" (UID: \"cc24a706-f079-4dce-97e5-95e7822ed722\") " Oct 30 22:17:06.267760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.267733 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca" pod "cc24a706-f079-4dce-97e5-95e7822ed722" (UID: "cc24a706-f079-4dce-97e5-95e7822ed722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:06.271019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.270993 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft" pod "cc24a706-f079-4dce-97e5-95e7822ed722" (UID: "cc24a706-f079-4dce-97e5-95e7822ed722"). InnerVolumeSpecName "kube-api-access-7fsft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:06.271202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.271182 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca" pod "cc24a706-f079-4dce-97e5-95e7822ed722" (UID: "cc24a706-f079-4dce-97e5-95e7822ed722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:06.375296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.375216 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fsft\" (UniqueName: \"kubernetes.io/projected/cc24a706-f079-4dce-97e5-95e7822ed722-kube-api-access-7fsft\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:06.378591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.375472 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:06.378591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.375501 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc24a706-f079-4dce-97e5-95e7822ed722-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:06.562631 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:06.562593 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8357d86_de78_4480_8d71_f6a3b9bdb81f.slice/crio-e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331 WatchSource:0}: Error finding container e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331: Status 404 returned error can't find the container with id e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331 Oct 30 22:17:06.562921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:06.562898 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks"] Oct 30 22:17:07.058147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.058103 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" event={"ID":"c9721985-ff96-4206-ab76-70cad8b4d602","Type":"ContainerStarted","Data":"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b"} Oct 30 22:17:07.059710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.059684 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h"] Oct 30 22:17:07.061728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.061694 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" event={"ID":"e8357d86-de78-4480-8d71-f6a3b9bdb81f","Type":"ContainerStarted","Data":"ca8a609433add8c46baa452d83ad6d76da12f6121709ece54a79e8aa89cef306"} Oct 30 22:17:07.061837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.061751 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" event={"ID":"e8357d86-de78-4480-8d71-f6a3b9bdb81f","Type":"ContainerStarted","Data":"e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331"} Oct 30 22:17:07.063285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.063263 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" event={"ID":"cc24a706-f079-4dce-97e5-95e7822ed722","Type":"ContainerDied","Data":"56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2"} Oct 30 22:17:07.063379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.063300 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c8c9410ac634ef5ceb64d4432e52459920a4787acd226e60416949068954d2" Oct 30 22:17:07.063379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.063352 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t" Oct 30 22:17:07.063742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.063721 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eadaf90-tvl2h"] Oct 30 22:17:07.215016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.214942 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podStartSLOduration=2.214922633 podStartE2EDuration="2.214922633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.207775102 +0000 UTC m=+1355.176594666" watchObservedRunningTime="2026-10-30 22:17:07.214922633 +0000 UTC m=+1355.183742199" Oct 30 22:17:07.381299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.381204 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" podStartSLOduration=2.381191022 podStartE2EDuration="2.381191022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.379923948 +0000 UTC m=+1355.348743512" watchObservedRunningTime="2026-10-30 22:17:07.381191022 +0000 UTC m=+1355.350010580" Oct 30 22:17:07.479565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.479528 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:07.481026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.481000 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:07.481153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:07.481056 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:08.072837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:08.072803 4292 generic.go:356] "Generic (PLEG): container finished" podID="e8357d86-de78-4480-8d71-f6a3b9bdb81f" containerID="ca8a609433add8c46baa452d83ad6d76da12f6121709ece54a79e8aa89cef306" exitCode=0 Oct 30 22:17:08.073229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:08.072884 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" event={"ID":"e8357d86-de78-4480-8d71-f6a3b9bdb81f","Type":"ContainerDied","Data":"ca8a609433add8c46baa452d83ad6d76da12f6121709ece54a79e8aa89cef306"} Oct 30 22:17:08.148713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:08.148684 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4514ae-7b5f-4382-a3a4-ce759d68b05c" path="/var/lib/kubelet/pods/da4514ae-7b5f-4382-a3a4-ce759d68b05c/volumes" Oct 30 22:17:08.479173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:08.479096 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:08.479173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:08.479153 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:09.337429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.337362 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:09.357390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.357361 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7"] Oct 30 22:17:09.358136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358118 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cc24a706-f079-4dce-97e5-95e7822ed722" containerName="revoke-keys" Oct 30 22:17:09.358186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358139 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc24a706-f079-4dce-97e5-95e7822ed722" containerName="revoke-keys" Oct 30 22:17:09.358186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358158 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e8357d86-de78-4480-8d71-f6a3b9bdb81f" containerName="revoke-keys" Oct 30 22:17:09.358186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358163 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8357d86-de78-4480-8d71-f6a3b9bdb81f" containerName="revoke-keys" Oct 30 22:17:09.358281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358263 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc24a706-f079-4dce-97e5-95e7822ed722" containerName="revoke-keys" Oct 30 22:17:09.358281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.358273 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8357d86-de78-4480-8d71-f6a3b9bdb81f" containerName="revoke-keys" Oct 30 22:17:09.361729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.361712 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.421456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.421415 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7"] Oct 30 22:17:09.449762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.449731 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\") pod \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " Oct 30 22:17:09.449933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.449789 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\") pod \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " Oct 30 22:17:09.449933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.449917 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\" (UniqueName: \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\") pod \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\" (UID: \"e8357d86-de78-4480-8d71-f6a3b9bdb81f\") " Oct 30 22:17:09.450177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.450155 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.450301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.450244 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca" pod "e8357d86-de78-4480-8d71-f6a3b9bdb81f" (UID: "e8357d86-de78-4480-8d71-f6a3b9bdb81f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:09.450482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.450441 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p885q\" (UniqueName: \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.450559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.450485 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.450655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.450632 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.465451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.465402 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca" pod "e8357d86-de78-4480-8d71-f6a3b9bdb81f" (UID: "e8357d86-de78-4480-8d71-f6a3b9bdb81f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:09.466234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.466192 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz" pod "e8357d86-de78-4480-8d71-f6a3b9bdb81f" (UID: "e8357d86-de78-4480-8d71-f6a3b9bdb81f"). InnerVolumeSpecName "kube-api-access-pd2xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.481621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.479414 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:09.481621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.479472 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:09.551810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.551771 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.552344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.552315 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p885q\" (UniqueName: \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.552495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.552478 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.552698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.552678 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8357d86-de78-4480-8d71-f6a3b9bdb81f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.552817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.552803 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2xz\" (UniqueName: \"kubernetes.io/projected/e8357d86-de78-4480-8d71-f6a3b9bdb81f-kube-api-access-pd2xz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.553382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.553356 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.555857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.555833 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.615456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.615385 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p885q\" (UniqueName: \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\") pod \"maas-api-revoke-sub-75dd5f5c-5pql7\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:09.674881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:09.674848 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:10.115600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.104647 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7"] Oct 30 22:17:10.122636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.119326 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" event={"ID":"e8357d86-de78-4480-8d71-f6a3b9bdb81f","Type":"ContainerDied","Data":"e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331"} Oct 30 22:17:10.122636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.119382 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e4af96b1942126f0f7226582179c9ac23def901c5d9aad9efcd5b4c14da331" Oct 30 22:17:10.122636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.119404 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks" Oct 30 22:17:10.479449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.479412 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:10.479831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:10.479470 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:11.127916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:11.127875 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" event={"ID":"79af5c0b-6de5-48fd-9190-5787e32c1971","Type":"ContainerStarted","Data":"b2e7b38c8a61b4fe29107c5814189feaeaa049abd3b254f746de2d0b985c586d"} Oct 30 22:17:11.128089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:11.127942 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" event={"ID":"79af5c0b-6de5-48fd-9190-5787e32c1971","Type":"ContainerStarted","Data":"9138b4540eefbf8180bad182656f397848615cf4510d6a4b7fe0665a036b068a"} Oct 30 22:17:11.249607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:11.249536 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" podStartSLOduration=2.24952217 podStartE2EDuration="2.24952217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:11.247450051 +0000 UTC m=+1359.216269628" watchObservedRunningTime="2026-10-30 22:17:11.24952217 +0000 UTC m=+1359.218341733" Oct 30 22:17:11.479113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:11.479023 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:11.479113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:11.479091 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:12.137264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:12.137232 4292 generic.go:356] "Generic (PLEG): container finished" podID="79af5c0b-6de5-48fd-9190-5787e32c1971" containerID="b2e7b38c8a61b4fe29107c5814189feaeaa049abd3b254f746de2d0b985c586d" exitCode=0 Oct 30 22:17:12.137670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:12.137266 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" event={"ID":"79af5c0b-6de5-48fd-9190-5787e32c1971","Type":"ContainerDied","Data":"b2e7b38c8a61b4fe29107c5814189feaeaa049abd3b254f746de2d0b985c586d"} Oct 30 22:17:12.479638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:12.479541 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:17:12.479638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:12.479605 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:17:13.327699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.327670 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:13.425418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.425350 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\" (UniqueName: \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\") pod \"79af5c0b-6de5-48fd-9190-5787e32c1971\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " Oct 30 22:17:13.425685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.425664 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\") pod \"79af5c0b-6de5-48fd-9190-5787e32c1971\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " Oct 30 22:17:13.425744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.425699 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\") pod \"79af5c0b-6de5-48fd-9190-5787e32c1971\" (UID: \"79af5c0b-6de5-48fd-9190-5787e32c1971\") " Oct 30 22:17:13.425957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.425937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca" pod "79af5c0b-6de5-48fd-9190-5787e32c1971" (UID: "79af5c0b-6de5-48fd-9190-5787e32c1971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:13.426222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.426207 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.427430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.427409 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q" pod "79af5c0b-6de5-48fd-9190-5787e32c1971" (UID: "79af5c0b-6de5-48fd-9190-5787e32c1971"). InnerVolumeSpecName "kube-api-access-p885q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:13.427602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.427566 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca" pod "79af5c0b-6de5-48fd-9190-5787e32c1971" (UID: "79af5c0b-6de5-48fd-9190-5787e32c1971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:13.486833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.486799 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:13.487646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.487594 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:13.488362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.488344 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:17:13.527323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.527293 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79af5c0b-6de5-48fd-9190-5787e32c1971-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.527434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:13.527348 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p885q\" (UniqueName: \"kubernetes.io/projected/79af5c0b-6de5-48fd-9190-5787e32c1971-kube-api-access-p885q\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:14.152799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.152761 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" Oct 30 22:17:14.152997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.152812 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7" event={"ID":"79af5c0b-6de5-48fd-9190-5787e32c1971","Type":"ContainerDied","Data":"9138b4540eefbf8180bad182656f397848615cf4510d6a4b7fe0665a036b068a"} Oct 30 22:17:14.152997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.152859 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9138b4540eefbf8180bad182656f397848615cf4510d6a4b7fe0665a036b068a" Oct 30 22:17:14.755738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.755704 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4"] Oct 30 22:17:14.756548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.756526 4292 cpu_manager.go:414] "containerMap: removing container" podUID="79af5c0b-6de5-48fd-9190-5787e32c1971" containerName="revoke-keys" Oct 30 22:17:14.756668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.756549 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79af5c0b-6de5-48fd-9190-5787e32c1971" containerName="revoke-keys" Oct 30 22:17:14.756742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.756716 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="79af5c0b-6de5-48fd-9190-5787e32c1971" containerName="revoke-keys" Oct 30 22:17:14.760382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.760361 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:14.815162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.815133 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:14.815353 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.815165 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:17:14.820462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.820438 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4"] Oct 30 22:17:14.903282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.903252 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:14.903417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.903298 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:14.947879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.947844 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjk79\" (UniqueName: \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:14.948013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.947959 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:14.948051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:14.948041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.048941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.048911 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjk79\" (UniqueName: \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.049088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.049012 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.049126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.049086 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.049893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.049870 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.051453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.051435 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.078634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.078606 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx"] Oct 30 22:17:15.082740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.082719 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93dc2fc1-sxscx"] Oct 30 22:17:15.113641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.113621 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjk79\" (UniqueName: \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\") pod \"maas-api-revoke-sub-88447a0b-hzms4\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.375650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.375565 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:15.782764 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:15.782735 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c2d253_b8fc_4635_94d7_088452af5b03.slice/crio-1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627 WatchSource:0}: Error finding container 1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627: Status 404 returned error can't find the container with id 1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627 Oct 30 22:17:15.783191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:15.782814 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4"] Oct 30 22:17:16.059840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.059809 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5"] Oct 30 22:17:16.069050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.069013 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e0de7f-j99r5"] Oct 30 22:17:16.147790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.147759 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a257b4d5-4829-485d-a0c1-5643772e4c23" path="/var/lib/kubelet/pods/a257b4d5-4829-485d-a0c1-5643772e4c23/volumes" Oct 30 22:17:16.148255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.148241 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d688ff4b-1b4b-4172-b542-c1bd6e06c179" path="/var/lib/kubelet/pods/d688ff4b-1b4b-4172-b542-c1bd6e06c179/volumes" Oct 30 22:17:16.169159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.169125 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" event={"ID":"e2c2d253-b8fc-4635-94d7-088452af5b03","Type":"ContainerStarted","Data":"5ec2ee5a35e91c51196f7ee14c6173505a22a74b45e7b8d07ebdce840402d57e"} Oct 30 22:17:16.169284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.169189 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" event={"ID":"e2c2d253-b8fc-4635-94d7-088452af5b03","Type":"ContainerStarted","Data":"1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627"} Oct 30 22:17:16.289243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:16.289162 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" podStartSLOduration=2.289146848 podStartE2EDuration="2.289146848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:16.287896471 +0000 UTC m=+1364.256716036" watchObservedRunningTime="2026-10-30 22:17:16.289146848 +0000 UTC m=+1364.257966428" Oct 30 22:17:17.178145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:17.178112 4292 generic.go:356] "Generic (PLEG): container finished" podID="e2c2d253-b8fc-4635-94d7-088452af5b03" containerID="5ec2ee5a35e91c51196f7ee14c6173505a22a74b45e7b8d07ebdce840402d57e" exitCode=0 Oct 30 22:17:17.178522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:17.178190 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" event={"ID":"e2c2d253-b8fc-4635-94d7-088452af5b03","Type":"ContainerDied","Data":"5ec2ee5a35e91c51196f7ee14c6173505a22a74b45e7b8d07ebdce840402d57e"} Oct 30 22:17:18.059508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.059471 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq"] Oct 30 22:17:18.109860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.109824 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8800e8c4-x96bq"] Oct 30 22:17:18.147902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.147872 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b6b58d-910f-4ebf-9632-acfcb35e667a" path="/var/lib/kubelet/pods/96b6b58d-910f-4ebf-9632-acfcb35e667a/volumes" Oct 30 22:17:18.349637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.349613 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:18.397737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.397713 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\" (UniqueName: \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\") pod \"e2c2d253-b8fc-4635-94d7-088452af5b03\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " Oct 30 22:17:18.397997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.397978 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\") pod \"e2c2d253-b8fc-4635-94d7-088452af5b03\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " Oct 30 22:17:18.398071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.398039 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\") pod \"e2c2d253-b8fc-4635-94d7-088452af5b03\" (UID: \"e2c2d253-b8fc-4635-94d7-088452af5b03\") " Oct 30 22:17:18.398314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.398293 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca" pod "e2c2d253-b8fc-4635-94d7-088452af5b03" (UID: "e2c2d253-b8fc-4635-94d7-088452af5b03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:18.398562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.398546 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:18.399841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.399819 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79" pod "e2c2d253-b8fc-4635-94d7-088452af5b03" (UID: "e2c2d253-b8fc-4635-94d7-088452af5b03"). InnerVolumeSpecName "kube-api-access-mjk79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:18.399978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.399959 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca" pod "e2c2d253-b8fc-4635-94d7-088452af5b03" (UID: "e2c2d253-b8fc-4635-94d7-088452af5b03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:18.500782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.500748 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2d253-b8fc-4635-94d7-088452af5b03-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:18.500782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.500785 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjk79\" (UniqueName: \"kubernetes.io/projected/e2c2d253-b8fc-4635-94d7-088452af5b03-kube-api-access-mjk79\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:18.599777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.599701 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:17:18.600292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.600278 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e2c2d253-b8fc-4635-94d7-088452af5b03" containerName="revoke-keys" Oct 30 22:17:18.600330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.600294 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c2d253-b8fc-4635-94d7-088452af5b03" containerName="revoke-keys" Oct 30 22:17:18.600425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.600415 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c2d253-b8fc-4635-94d7-088452af5b03" containerName="revoke-keys" Oct 30 22:17:18.604021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.604001 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.670501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.670469 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:17:18.702804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.702773 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.702993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.702821 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.702993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.702977 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703072 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703197 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703234 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703372 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ghz\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.703461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.703419 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.751061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.751018 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn"] Oct 30 22:17:18.761604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.755699 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.804666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804635 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.804846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804712 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.804846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804741 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qgb\" (UniqueName: \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.804846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804802 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804909 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.804981 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.805293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805265 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805406 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805421 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ghz\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805515 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805609 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.805829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805649 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.806267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.806267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.805430 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.806267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.806230 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.806474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.806378 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.808226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.808201 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.811406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.811385 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.817330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.817305 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn"] Oct 30 22:17:18.869598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.869499 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ghz\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.869598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.869556 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.906740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.906711 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qgb\" (UniqueName: \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.906894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.906772 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.906963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.906949 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.907396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.907371 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.909086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.909067 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:18.918121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.918099 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:18.970429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:18.970395 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qgb\" (UniqueName: \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\") pod \"maas-api-revoke-sub-f2812b60-nnhrn\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:19.058075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.058042 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8"] Oct 30 22:17:19.062040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.062019 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49efaede-gpgk8"] Oct 30 22:17:19.073690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.073668 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:19.201310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.201266 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" event={"ID":"e2c2d253-b8fc-4635-94d7-088452af5b03","Type":"ContainerDied","Data":"1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627"} Oct 30 22:17:19.201473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.201340 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1123ba00a8cc42173b73bd26d2cf44b6efc102c8fcdeacafc966b432bd985627" Oct 30 22:17:19.201473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.201284 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4" Oct 30 22:17:19.353886 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:19.353855 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b68b888_0376_4a8f_8c03_c984fad1f205.slice/crio-a4f66fcbc5e60d62e93920e1ba89883bc209beacca8991cef32fb806aa822236 WatchSource:0}: Error finding container a4f66fcbc5e60d62e93920e1ba89883bc209beacca8991cef32fb806aa822236: Status 404 returned error can't find the container with id a4f66fcbc5e60d62e93920e1ba89883bc209beacca8991cef32fb806aa822236 Oct 30 22:17:19.355823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.355802 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:17:19.356666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.356635 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:19.356766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.356697 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:19.356766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.356728 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:19.461617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:19.457392 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn"] Oct 30 22:17:20.078499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.078467 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r"] Oct 30 22:17:20.083145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.083114 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.125437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.125404 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.125437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.125445 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.125642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.125583 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n762j\" (UniqueName: \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.152528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.152495 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdb1fc5-45a9-4f6d-ab34-078c88e59025" path="/var/lib/kubelet/pods/ffdb1fc5-45a9-4f6d-ab34-078c88e59025/volumes" Oct 30 22:17:20.153012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.152991 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r"] Oct 30 22:17:20.220861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.220816 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" event={"ID":"70582417-c421-44cf-bd1d-284ccda74916","Type":"ContainerStarted","Data":"ec77fdf9a04da799a4144c28d241776d1daee45a2893adbd8434b0c253200e51"} Oct 30 22:17:20.221058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.220877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" event={"ID":"70582417-c421-44cf-bd1d-284ccda74916","Type":"ContainerStarted","Data":"97b0eea3fcf7c0e9bdee5c60ce5dd83412ac494c7f0952ae0cbbcb27de72b890"} Oct 30 22:17:20.222602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.222554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" event={"ID":"0b68b888-0376-4a8f-8c03-c984fad1f205","Type":"ContainerStarted","Data":"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da"} Oct 30 22:17:20.222726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.222617 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" event={"ID":"0b68b888-0376-4a8f-8c03-c984fad1f205","Type":"ContainerStarted","Data":"a4f66fcbc5e60d62e93920e1ba89883bc209beacca8991cef32fb806aa822236"} Oct 30 22:17:20.226935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.226914 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n762j\" (UniqueName: \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.227244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.227214 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.227362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.227284 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.227925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.227902 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.229509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.229490 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.306601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.302715 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n762j\" (UniqueName: \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\") pod \"maas-api-revoke-sub-0d536d07-6rq8r\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.341634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.341515 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" podStartSLOduration=2.341495688 podStartE2EDuration="2.341495688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:20.338839641 +0000 UTC m=+1368.307659206" watchObservedRunningTime="2026-10-30 22:17:20.341495688 +0000 UTC m=+1368.310315251" Oct 30 22:17:20.421032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.421000 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:20.482936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.479722 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podStartSLOduration=2.479707507 podStartE2EDuration="2.479707507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:20.479021295 +0000 UTC m=+1368.447840859" watchObservedRunningTime="2026-10-30 22:17:20.479707507 +0000 UTC m=+1368.448527071" Oct 30 22:17:20.815047 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:20.815009 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6923b1bc_46ba_43d1_8f12_d011884089a0.slice/crio-36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac WatchSource:0}: Error finding container 36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac: Status 404 returned error can't find the container with id 36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac Oct 30 22:17:20.815237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.815213 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r"] Oct 30 22:17:20.918934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.918902 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:20.920467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.920440 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:20.920603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:20.920489 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:21.231722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.231689 4292 generic.go:356] "Generic (PLEG): container finished" podID="70582417-c421-44cf-bd1d-284ccda74916" containerID="ec77fdf9a04da799a4144c28d241776d1daee45a2893adbd8434b0c253200e51" exitCode=0 Oct 30 22:17:21.231834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.231776 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" event={"ID":"70582417-c421-44cf-bd1d-284ccda74916","Type":"ContainerDied","Data":"ec77fdf9a04da799a4144c28d241776d1daee45a2893adbd8434b0c253200e51"} Oct 30 22:17:21.233550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.233526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" event={"ID":"6923b1bc-46ba-43d1-8f12-d011884089a0","Type":"ContainerStarted","Data":"7349dda278dcd87008698b0cf68afcc70d9ded8e0cea5d1d80cb91344cfc4755"} Oct 30 22:17:21.233683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.233570 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" event={"ID":"6923b1bc-46ba-43d1-8f12-d011884089a0","Type":"ContainerStarted","Data":"36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac"} Oct 30 22:17:21.477984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.477886 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" podStartSLOduration=1.477868704 podStartE2EDuration="1.477868704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:21.475565165 +0000 UTC m=+1369.444384729" watchObservedRunningTime="2026-10-30 22:17:21.477868704 +0000 UTC m=+1369.446688316" Oct 30 22:17:21.919172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.919137 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:21.919356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:21.919193 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:22.243297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.243206 4292 generic.go:356] "Generic (PLEG): container finished" podID="6923b1bc-46ba-43d1-8f12-d011884089a0" containerID="7349dda278dcd87008698b0cf68afcc70d9ded8e0cea5d1d80cb91344cfc4755" exitCode=0 Oct 30 22:17:22.243297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.243281 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" event={"ID":"6923b1bc-46ba-43d1-8f12-d011884089a0","Type":"ContainerDied","Data":"7349dda278dcd87008698b0cf68afcc70d9ded8e0cea5d1d80cb91344cfc4755"} Oct 30 22:17:22.427110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.427085 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:22.460496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.460465 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\") pod \"70582417-c421-44cf-bd1d-284ccda74916\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " Oct 30 22:17:22.460675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.460649 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\") pod \"70582417-c421-44cf-bd1d-284ccda74916\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " Oct 30 22:17:22.460722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.460708 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\" (UniqueName: \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\") pod \"70582417-c421-44cf-bd1d-284ccda74916\" (UID: \"70582417-c421-44cf-bd1d-284ccda74916\") " Oct 30 22:17:22.460827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.460804 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca" pod "70582417-c421-44cf-bd1d-284ccda74916" (UID: "70582417-c421-44cf-bd1d-284ccda74916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:22.461190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.461170 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70582417-c421-44cf-bd1d-284ccda74916-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:22.462775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.462747 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb" pod "70582417-c421-44cf-bd1d-284ccda74916" (UID: "70582417-c421-44cf-bd1d-284ccda74916"). InnerVolumeSpecName "kube-api-access-57qgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:22.462775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.462750 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca" pod "70582417-c421-44cf-bd1d-284ccda74916" (UID: "70582417-c421-44cf-bd1d-284ccda74916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:22.562831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.562802 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70582417-c421-44cf-bd1d-284ccda74916-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:22.562831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.562831 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57qgb\" (UniqueName: \"kubernetes.io/projected/70582417-c421-44cf-bd1d-284ccda74916-kube-api-access-57qgb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:22.919159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.919083 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:22.919159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:22.919137 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:23.182521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.182439 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd"] Oct 30 22:17:23.183103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.183085 4292 cpu_manager.go:414] "containerMap: removing container" podUID="70582417-c421-44cf-bd1d-284ccda74916" containerName="revoke-keys" Oct 30 22:17:23.183103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.183102 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70582417-c421-44cf-bd1d-284ccda74916" containerName="revoke-keys" Oct 30 22:17:23.183236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.183208 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="70582417-c421-44cf-bd1d-284ccda74916" containerName="revoke-keys" Oct 30 22:17:23.187616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.186899 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.250713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.250679 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd"] Oct 30 22:17:23.255317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.255286 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" Oct 30 22:17:23.255677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.255653 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn" event={"ID":"70582417-c421-44cf-bd1d-284ccda74916","Type":"ContainerDied","Data":"97b0eea3fcf7c0e9bdee5c60ce5dd83412ac494c7f0952ae0cbbcb27de72b890"} Oct 30 22:17:23.256412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.256275 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b0eea3fcf7c0e9bdee5c60ce5dd83412ac494c7f0952ae0cbbcb27de72b890" Oct 30 22:17:23.270622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.270600 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.270742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.270644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.270742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.270696 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4gn\" (UniqueName: \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.374603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.373524 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4gn\" (UniqueName: \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.374603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.373833 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.374603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.373902 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.375301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.375276 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.416624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.415299 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.437936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.437872 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4gn\" (UniqueName: \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\") pod \"maas-api-revoke-sub-62702043-8txmd\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.473671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.473645 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:23.521860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.521828 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:23.576601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.576530 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\") pod \"6923b1bc-46ba-43d1-8f12-d011884089a0\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " Oct 30 22:17:23.577647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.576616 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\" (UniqueName: \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\") pod \"6923b1bc-46ba-43d1-8f12-d011884089a0\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " Oct 30 22:17:23.577647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.576702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\") pod \"6923b1bc-46ba-43d1-8f12-d011884089a0\" (UID: \"6923b1bc-46ba-43d1-8f12-d011884089a0\") " Oct 30 22:17:23.577811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.577716 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca" pod "6923b1bc-46ba-43d1-8f12-d011884089a0" (UID: "6923b1bc-46ba-43d1-8f12-d011884089a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:23.580627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.580599 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca" pod "6923b1bc-46ba-43d1-8f12-d011884089a0" (UID: "6923b1bc-46ba-43d1-8f12-d011884089a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:23.580951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.580922 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j" pod "6923b1bc-46ba-43d1-8f12-d011884089a0" (UID: "6923b1bc-46ba-43d1-8f12-d011884089a0"). InnerVolumeSpecName "kube-api-access-n762j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:23.635199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.633844 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv"] Oct 30 22:17:23.635199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.634956 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6923b1bc-46ba-43d1-8f12-d011884089a0" containerName="revoke-keys" Oct 30 22:17:23.635199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.634977 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6923b1bc-46ba-43d1-8f12-d011884089a0" containerName="revoke-keys" Oct 30 22:17:23.635199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.635163 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6923b1bc-46ba-43d1-8f12-d011884089a0" containerName="revoke-keys" Oct 30 22:17:23.642113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.641825 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679261 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd46z\" (UniqueName: \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679360 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679436 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679586 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679603 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n762j\" (UniqueName: \"kubernetes.io/projected/6923b1bc-46ba-43d1-8f12-d011884089a0-kube-api-access-n762j\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.680150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.679615 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6923b1bc-46ba-43d1-8f12-d011884089a0-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.718890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.718806 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv"] Oct 30 22:17:23.781119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.781080 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd46z\" (UniqueName: \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.781357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.781262 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.781436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.781357 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.782106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.782081 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.783989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.783964 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.844752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.844723 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd46z\" (UniqueName: \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\") pod \"maas-api-revoke-sub-73a89c8a-8l9wv\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:23.914496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.914465 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd"] Oct 30 22:17:23.914666 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:23.914642 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c973869_5d74_4b8e_9a37_8226029ed950.slice/crio-dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725 WatchSource:0}: Error finding container dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725: Status 404 returned error can't find the container with id dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725 Oct 30 22:17:23.919134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.919111 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:23.919231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.919152 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:23.956912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:23.956890 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:24.264474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.264390 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" event={"ID":"6923b1bc-46ba-43d1-8f12-d011884089a0","Type":"ContainerDied","Data":"36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac"} Oct 30 22:17:24.264474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.264443 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a1fb342600cb2dbc0494213e6830ea5a288b102ab9a93577814253e17d80ac" Oct 30 22:17:24.264717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.264509 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r" Oct 30 22:17:24.266308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.266246 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" event={"ID":"0c973869-5d74-4b8e-9a37-8226029ed950","Type":"ContainerStarted","Data":"dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725"} Oct 30 22:17:24.465365 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:24.465334 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee398022_de46_4098_8027_6af20f8d0ef0.slice/crio-acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1 WatchSource:0}: Error finding container acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1: Status 404 returned error can't find the container with id acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1 Oct 30 22:17:24.465555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.465502 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv"] Oct 30 22:17:24.859865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.859830 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j"] Oct 30 22:17:24.866756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.866729 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:24.918883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.918849 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j"] Oct 30 22:17:24.923414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.923278 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:24.923414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:24.923328 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:25.026896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.026858 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479kl\" (UniqueName: \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.027099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.027002 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.027099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.027045 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.128105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.128033 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kl\" (UniqueName: \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.128234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.128111 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.128234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.128139 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.128975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.128954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.130487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.130469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.211327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.211291 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479kl\" (UniqueName: \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\") pod \"maas-api-revoke-sub-bf6fb3b8-2qf2j\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.277408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.277372 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" event={"ID":"ee398022-de46-4098-8027-6af20f8d0ef0","Type":"ContainerStarted","Data":"2a6fbbc2d5c699301734b976cfb028fb4e6f8c00c1f6e7a3bfee8e9c066e84d0"} Oct 30 22:17:25.277604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.277423 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" event={"ID":"ee398022-de46-4098-8027-6af20f8d0ef0","Type":"ContainerStarted","Data":"acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1"} Oct 30 22:17:25.279054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.279025 4292 generic.go:356] "Generic (PLEG): container finished" podID="0c973869-5d74-4b8e-9a37-8226029ed950" containerID="52377c8232c2b732fa9e787a13d66967edd3b4b50f6cf554e2d70708ea2e183e" exitCode=0 Oct 30 22:17:25.279160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.279107 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" event={"ID":"0c973869-5d74-4b8e-9a37-8226029ed950","Type":"ContainerDied","Data":"52377c8232c2b732fa9e787a13d66967edd3b4b50f6cf554e2d70708ea2e183e"} Oct 30 22:17:25.403221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.403104 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" podStartSLOduration=2.403085643 podStartE2EDuration="2.403085643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:25.398319329 +0000 UTC m=+1373.367138893" watchObservedRunningTime="2026-10-30 22:17:25.403085643 +0000 UTC m=+1373.371905208" Oct 30 22:17:25.483685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.483653 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:25.867567 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:25.867528 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09fd51b_2e4a_403d_8a67_ebc8836f3fc5.slice/crio-c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444 WatchSource:0}: Error finding container c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444: Status 404 returned error can't find the container with id c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444 Oct 30 22:17:25.867952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.867807 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j"] Oct 30 22:17:25.918597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.918555 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:17:25.918702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:25.918630 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:17:26.288562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.288514 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" event={"ID":"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5","Type":"ContainerStarted","Data":"ae4d2a46f3565f92bb4970b414b4fae023e7a7bbe6309896e3b7b8b145e80ee6"} Oct 30 22:17:26.288749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.288619 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" event={"ID":"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5","Type":"ContainerStarted","Data":"c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444"} Oct 30 22:17:26.290739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.290711 4292 generic.go:356] "Generic (PLEG): container finished" podID="ee398022-de46-4098-8027-6af20f8d0ef0" containerID="2a6fbbc2d5c699301734b976cfb028fb4e6f8c00c1f6e7a3bfee8e9c066e84d0" exitCode=0 Oct 30 22:17:26.290858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.290792 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" event={"ID":"ee398022-de46-4098-8027-6af20f8d0ef0","Type":"ContainerDied","Data":"2a6fbbc2d5c699301734b976cfb028fb4e6f8c00c1f6e7a3bfee8e9c066e84d0"} Oct 30 22:17:26.408139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.408078 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" podStartSLOduration=2.40805936 podStartE2EDuration="2.40805936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:26.407631262 +0000 UTC m=+1374.376450827" watchObservedRunningTime="2026-10-30 22:17:26.40805936 +0000 UTC m=+1374.376878928" Oct 30 22:17:26.475375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.475346 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:26.548019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.547980 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\") pod \"0c973869-5d74-4b8e-9a37-8226029ed950\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " Oct 30 22:17:26.548195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.548034 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\" (UniqueName: \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\") pod \"0c973869-5d74-4b8e-9a37-8226029ed950\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " Oct 30 22:17:26.548195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.548113 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\") pod \"0c973869-5d74-4b8e-9a37-8226029ed950\" (UID: \"0c973869-5d74-4b8e-9a37-8226029ed950\") " Oct 30 22:17:26.548507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.548471 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca" pod "0c973869-5d74-4b8e-9a37-8226029ed950" (UID: "0c973869-5d74-4b8e-9a37-8226029ed950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:26.548839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.548817 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.550202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.550173 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca" pod "0c973869-5d74-4b8e-9a37-8226029ed950" (UID: "0c973869-5d74-4b8e-9a37-8226029ed950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:26.550670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.550647 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn" pod "0c973869-5d74-4b8e-9a37-8226029ed950" (UID: "0c973869-5d74-4b8e-9a37-8226029ed950"). InnerVolumeSpecName "kube-api-access-xm4gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:26.642933 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:17:26.642893 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:17:26.650538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.650505 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c973869-5d74-4b8e-9a37-8226029ed950-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.650538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.650539 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4gn\" (UniqueName: \"kubernetes.io/projected/0c973869-5d74-4b8e-9a37-8226029ed950-kube-api-access-xm4gn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.922804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.922776 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:26.923269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.923256 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:26.923950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:26.923925 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:17:27.306852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.306812 4292 generic.go:356] "Generic (PLEG): container finished" podID="f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" containerID="ae4d2a46f3565f92bb4970b414b4fae023e7a7bbe6309896e3b7b8b145e80ee6" exitCode=0 Oct 30 22:17:27.307058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.306897 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" event={"ID":"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5","Type":"ContainerDied","Data":"ae4d2a46f3565f92bb4970b414b4fae023e7a7bbe6309896e3b7b8b145e80ee6"} Oct 30 22:17:27.308563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.308543 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" Oct 30 22:17:27.308676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.308592 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd" event={"ID":"0c973869-5d74-4b8e-9a37-8226029ed950","Type":"ContainerDied","Data":"dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725"} Oct 30 22:17:27.308676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.308634 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfaa2366dd845bce75c280371536ca7667369b1aac4bc8aa9c88c63115d1725" Oct 30 22:17:27.465144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.465120 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:27.562517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.562438 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\") pod \"ee398022-de46-4098-8027-6af20f8d0ef0\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " Oct 30 22:17:27.562718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.562601 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\" (UniqueName: \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\") pod \"ee398022-de46-4098-8027-6af20f8d0ef0\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " Oct 30 22:17:27.562804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.562753 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\") pod \"ee398022-de46-4098-8027-6af20f8d0ef0\" (UID: \"ee398022-de46-4098-8027-6af20f8d0ef0\") " Oct 30 22:17:27.562888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.562865 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca" pod "ee398022-de46-4098-8027-6af20f8d0ef0" (UID: "ee398022-de46-4098-8027-6af20f8d0ef0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:27.563307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.563278 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.564614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.564563 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z" pod "ee398022-de46-4098-8027-6af20f8d0ef0" (UID: "ee398022-de46-4098-8027-6af20f8d0ef0"). InnerVolumeSpecName "kube-api-access-jd46z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:27.564697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.564647 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca" pod "ee398022-de46-4098-8027-6af20f8d0ef0" (UID: "ee398022-de46-4098-8027-6af20f8d0ef0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:27.665099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.665066 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd46z\" (UniqueName: \"kubernetes.io/projected/ee398022-de46-4098-8027-6af20f8d0ef0-kube-api-access-jd46z\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.665099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:27.665097 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee398022-de46-4098-8027-6af20f8d0ef0-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.318941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.318915 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" Oct 30 22:17:28.319356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.318959 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv" event={"ID":"ee398022-de46-4098-8027-6af20f8d0ef0","Type":"ContainerDied","Data":"acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1"} Oct 30 22:17:28.319356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.319009 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acedf31c1bcedd58a21426afb275d34750832bf48a72677160e03ee0f4a961e1" Oct 30 22:17:28.473089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.473062 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:28.579262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.579232 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\") pod \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " Oct 30 22:17:28.579432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.579277 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\") pod \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " Oct 30 22:17:28.579432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.579368 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\" (UniqueName: \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\") pod \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\" (UID: \"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5\") " Oct 30 22:17:28.579679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.579648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca" pod "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" (UID: "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:28.579897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.579871 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.581458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.581439 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca" pod "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" (UID: "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:28.581458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.581447 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl" pod "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" (UID: "f09fd51b-2e4a-403d-8a67-ebc8836f3fc5"). InnerVolumeSpecName "kube-api-access-479kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:28.681231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.681199 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.681231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:28.681229 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479kl\" (UniqueName: \"kubernetes.io/projected/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5-kube-api-access-479kl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:29.328768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:29.328736 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" Oct 30 22:17:29.329134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:29.328771 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j" event={"ID":"f09fd51b-2e4a-403d-8a67-ebc8836f3fc5","Type":"ContainerDied","Data":"c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444"} Oct 30 22:17:29.329134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:29.328817 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c912076aa4a2448b3738350d6bd19c6dbe9549c6a132b5542ebaea0419066444" Oct 30 22:17:30.057568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.057535 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg"] Oct 30 22:17:30.058423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058403 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" containerName="revoke-keys" Oct 30 22:17:30.058518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058424 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" containerName="revoke-keys" Oct 30 22:17:30.058518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058439 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0c973869-5d74-4b8e-9a37-8226029ed950" containerName="revoke-keys" Oct 30 22:17:30.058518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058444 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c973869-5d74-4b8e-9a37-8226029ed950" containerName="revoke-keys" Oct 30 22:17:30.058518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058455 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ee398022-de46-4098-8027-6af20f8d0ef0" containerName="revoke-keys" Oct 30 22:17:30.058518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058461 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee398022-de46-4098-8027-6af20f8d0ef0" containerName="revoke-keys" Oct 30 22:17:30.058793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058564 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" containerName="revoke-keys" Oct 30 22:17:30.058793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058585 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c973869-5d74-4b8e-9a37-8226029ed950" containerName="revoke-keys" Oct 30 22:17:30.058793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.058597 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee398022-de46-4098-8027-6af20f8d0ef0" containerName="revoke-keys" Oct 30 22:17:30.062131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.062110 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.065215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.065190 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb"] Oct 30 22:17:30.107674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.107646 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf795471-gmmmb"] Oct 30 22:17:30.116543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.116519 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:30.121623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.121567 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg"] Oct 30 22:17:30.149366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.149337 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0e9621-1b6f-488f-8a53-ae21948ab815" path="/var/lib/kubelet/pods/5e0e9621-1b6f-488f-8a53-ae21948ab815/volumes" Oct 30 22:17:30.153423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.153402 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:30.153540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.153467 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:17:30.153540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.153469 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:30.203224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.203193 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.203355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.203259 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.203425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.203367 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5pp\" (UniqueName: \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.304413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.304382 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.304637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.304444 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.304637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.304492 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5pp\" (UniqueName: \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.305222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.305203 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.306816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.306783 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.367649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.367585 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5pp\" (UniqueName: \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\") pod \"maas-api-revoke-sub-c393eab2-zmtwg\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.377430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.377405 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:30.774744 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:30.774681 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014fd968_cad3_48b0_8622_0700478b90af.slice/crio-76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742 WatchSource:0}: Error finding container 76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742: Status 404 returned error can't find the container with id 76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742 Oct 30 22:17:30.777415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:30.777378 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg"] Oct 30 22:17:31.437606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:31.427775 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" event={"ID":"014fd968-cad3-48b0-8622-0700478b90af","Type":"ContainerStarted","Data":"de875bc2b4b48038cb312559cbeb5112ddb1b13cd8a1571d5043bbfea1a5eb40"} Oct 30 22:17:31.437606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:31.427861 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" event={"ID":"014fd968-cad3-48b0-8622-0700478b90af","Type":"ContainerStarted","Data":"76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742"} Oct 30 22:17:31.582602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:31.582449 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" podStartSLOduration=1.5824305239999998 podStartE2EDuration="1.582430524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:31.576949229 +0000 UTC m=+1379.545768794" watchObservedRunningTime="2026-10-30 22:17:31.582430524 +0000 UTC m=+1379.551250089" Oct 30 22:17:32.474595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.472974 4292 scope.go:122] "RemoveContainer" containerID="5a7fdbe879e74518e04eb003520ba7ef4d0a9efa94c12065fe9e04f1cfbadcc7" Oct 30 22:17:32.508267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.504649 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj"] Oct 30 22:17:32.523467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.523432 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8a213-dn5kj"] Oct 30 22:17:32.597027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.596928 4292 scope.go:122] "RemoveContainer" containerID="90664f32c47218936a20ae384903cb19e6874b349eb7e54256901505817f1af4" Oct 30 22:17:32.600234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.599622 4292 generic.go:356] "Generic (PLEG): container finished" podID="014fd968-cad3-48b0-8622-0700478b90af" containerID="de875bc2b4b48038cb312559cbeb5112ddb1b13cd8a1571d5043bbfea1a5eb40" exitCode=0 Oct 30 22:17:32.600234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.599775 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" event={"ID":"014fd968-cad3-48b0-8622-0700478b90af","Type":"ContainerDied","Data":"de875bc2b4b48038cb312559cbeb5112ddb1b13cd8a1571d5043bbfea1a5eb40"} Oct 30 22:17:32.779626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.766919 4292 scope.go:122] "RemoveContainer" containerID="812b3ec16fc6407ba1fc285cdb1ca7c09c5695a6694b136dedde197b611482a7" Oct 30 22:17:32.779626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.771886 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:32.889601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.885909 4292 scope.go:122] "RemoveContainer" containerID="c517750a472b5aada2935167017020ed3d7fa645e1529510b9a9d2496a3446ae" Oct 30 22:17:32.990628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:32.981714 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.015649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.014800 4292 scope.go:122] "RemoveContainer" containerID="2475cf342a982b8b98e24765da2cbd4f2ef3ce5b6fd6f37773fbec7220e9331c" Oct 30 22:17:33.039819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.033678 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:17:33.039819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.034373 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.112404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.089442 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:17:33.208673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.199316 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.288679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.248983 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.288679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.263614 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9"] Oct 30 22:17:33.307612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.304651 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-854f9f49-cd4s9"] Oct 30 22:17:33.311704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.311652 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.718146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.716212 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.809950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.809669 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:33.914009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.810129 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:17:33.914009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:33.844941 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.027705 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.027976 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.054114 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.055151 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.057255 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.058393 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:17:34.080652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.058475 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:17:34.108599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.093786 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:17:34.108599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.094163 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.213086 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.220268 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.223093 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.224482 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.249973 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.250147 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262374 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262449 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262522 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262686 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262769 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.262846 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghx9\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.263017 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.263070 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.353731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.263239 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.375288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.374876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.375288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375003 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.375288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375088 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.375288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375141 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.375288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375243 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.376182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375808 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.376182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.375885 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghx9\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.376182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.376027 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.376182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.376073 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.428778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.381731 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.428778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.405191 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.428778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.405632 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.428778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.406197 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.428778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.406726 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.521877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.491460 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:34.765094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.761807 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 22:17:34.885754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.861560 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:34.912742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.909475 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:34.912742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:34.910848 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:17:35.087135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.032654 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.087135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.067724 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.087135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.068042 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.087135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.068259 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:17:35.109732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.088849 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.109732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.095757 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:17:35.109732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.101707 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.210606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.204679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:35.288692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.274867 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:17:35.323975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.300352 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:35.323975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.300901 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:35.353091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.346981 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cacc12-d3e6-473b-af98-14e9500e9a0f" path="/var/lib/kubelet/pods/15cacc12-d3e6-473b-af98-14e9500e9a0f/volumes" Oct 30 22:17:35.378449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.378361 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghx9\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\") pod \"e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.398934 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.399803 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf7e9cd-ef91-4467-8e0f-09366f3f9a92" path="/var/lib/kubelet/pods/ecf7e9cd-ef91-4467-8e0f-09366f3f9a92/volumes" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:17:35.400378 4292 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.146s" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.400506 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.400552 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.400594 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:17:35.408720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.400655 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:17:35.525600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.516692 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.029599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:35.975045 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.071279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.039672 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.071279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.040114 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.181918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.123747 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.181918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.124186 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.181918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.137286 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:17:36.181918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.165389 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.181918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.173733 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.190970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.190900 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.275916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.275881 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:36.333993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.333846 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:17:36.388282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.388229 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:17:36.421307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.393910 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:17:36.421307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.393965 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:17:36.422150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.421877 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv"] Oct 30 22:17:36.422362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.422344 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc892c3-z6zlv"] Oct 30 22:17:36.423045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.423013 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:17:36.426162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.426138 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:17:36.430236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.430214 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:17:36.430405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.430390 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:17:36.430541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.430529 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:17:36.449587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.434393 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:17:36.465254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.465174 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt"] Oct 30 22:17:36.482313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.481106 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:17:36.482313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.481853 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.488996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.488966 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt"] Oct 30 22:17:36.522224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.518761 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:36.566267 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:36.566236 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35b8b33_0e4d_43ee_8242_fd41159c9dab.slice/crio-6ab132fcb0eadd4b87933aa583eb821c2b6450c522db8f95b2d571e9322ebb81 WatchSource:0}: Error finding container 6ab132fcb0eadd4b87933aa583eb821c2b6450c522db8f95b2d571e9322ebb81: Status 404 returned error can't find the container with id 6ab132fcb0eadd4b87933aa583eb821c2b6450c522db8f95b2d571e9322ebb81 Oct 30 22:17:36.566719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.566695 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:17:36.569254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.569224 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:36.569351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.569292 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:36.569351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.569326 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:36.581916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.581893 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.582478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.582441 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.582656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.582630 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kct2j\" (UniqueName: \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.684206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.684170 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\") pod \"014fd968-cad3-48b0-8622-0700478b90af\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " Oct 30 22:17:36.684482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.684443 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\") pod \"014fd968-cad3-48b0-8622-0700478b90af\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " Oct 30 22:17:36.684642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.684545 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca" pod "014fd968-cad3-48b0-8622-0700478b90af" (UID: "014fd968-cad3-48b0-8622-0700478b90af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:36.685396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.685370 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\" (UniqueName: \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\") pod \"014fd968-cad3-48b0-8622-0700478b90af\" (UID: \"014fd968-cad3-48b0-8622-0700478b90af\") " Oct 30 22:17:36.686308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.686281 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.687492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.687171 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.687824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.687802 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kct2j\" (UniqueName: \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.688684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.688540 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca" pod "014fd968-cad3-48b0-8622-0700478b90af" (UID: "014fd968-cad3-48b0-8622-0700478b90af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:36.688804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.688538 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp" pod "014fd968-cad3-48b0-8622-0700478b90af" (UID: "014fd968-cad3-48b0-8622-0700478b90af"). InnerVolumeSpecName "kube-api-access-zs5pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:36.688895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.688862 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.689022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.689006 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014fd968-cad3-48b0-8622-0700478b90af-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.689109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.689026 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5pp\" (UniqueName: \"kubernetes.io/projected/014fd968-cad3-48b0-8622-0700478b90af-kube-api-access-zs5pp\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.689109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.689042 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014fd968-cad3-48b0-8622-0700478b90af-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.691951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.691928 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.751700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.751669 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kct2j\" (UniqueName: \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\") pod \"maas-api-revoke-sub-9a535060-qqhxt\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:36.827744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:36.827711 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:37.070355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.070316 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s"] Oct 30 22:17:37.078950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.078863 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30b8795-tsw6s"] Oct 30 22:17:37.215760 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:37.215719 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d6b4ca_3688_49db_acbc_02a111d043cc.slice/crio-fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1 WatchSource:0}: Error finding container fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1: Status 404 returned error can't find the container with id fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1 Oct 30 22:17:37.215965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.215941 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt"] Oct 30 22:17:37.312324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.312285 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" event={"ID":"d35b8b33-0e4d-43ee-8242-fd41159c9dab","Type":"ContainerStarted","Data":"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278"} Oct 30 22:17:37.312610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.312550 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" event={"ID":"d35b8b33-0e4d-43ee-8242-fd41159c9dab","Type":"ContainerStarted","Data":"6ab132fcb0eadd4b87933aa583eb821c2b6450c522db8f95b2d571e9322ebb81"} Oct 30 22:17:37.313826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.313796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" event={"ID":"29d6b4ca-3688-49db-acbc-02a111d043cc","Type":"ContainerStarted","Data":"fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1"} Oct 30 22:17:37.315532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.315510 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" event={"ID":"014fd968-cad3-48b0-8622-0700478b90af","Type":"ContainerDied","Data":"76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742"} Oct 30 22:17:37.315682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.315548 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d0187ec1f127f193bed16e258f2386373d91f33f45dde13fb7f43aec3a7742" Oct 30 22:17:37.315682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.315566 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg" Oct 30 22:17:37.406051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.402773 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:37.406051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.404367 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:37.406051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.404418 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:37.925748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.925708 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg"] Oct 30 22:17:37.926797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.926774 4292 cpu_manager.go:414] "containerMap: removing container" podUID="014fd968-cad3-48b0-8622-0700478b90af" containerName="revoke-keys" Oct 30 22:17:37.926797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.926797 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014fd968-cad3-48b0-8622-0700478b90af" containerName="revoke-keys" Oct 30 22:17:37.926983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.926931 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="014fd968-cad3-48b0-8622-0700478b90af" containerName="revoke-keys" Oct 30 22:17:37.930562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.930542 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:37.974659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:37.974628 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg"] Oct 30 22:17:38.021106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.020487 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592bt\" (UniqueName: \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.021418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.021378 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.021725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.021702 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.122544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.122511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592bt\" (UniqueName: \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.123075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.122632 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.123075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.122736 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.123311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.123291 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.125079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.125061 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.150374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.150335 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ee8525-eafb-462b-b69b-78937278eae4" path="/var/lib/kubelet/pods/d1ee8525-eafb-462b-b69b-78937278eae4/volumes" Oct 30 22:17:38.151036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.151011 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea490d2-eb49-4d65-a661-70658bedde53" path="/var/lib/kubelet/pods/dea490d2-eb49-4d65-a661-70658bedde53/volumes" Oct 30 22:17:38.214906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.214369 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592bt\" (UniqueName: \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\") pod \"maas-api-revoke-sub-235bf2b5-npljg\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.246007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.245978 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:38.328942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.328899 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" event={"ID":"29d6b4ca-3688-49db-acbc-02a111d043cc","Type":"ContainerStarted","Data":"d9b8177d7033d96a5b57334ee7c10f53aa2803110b85fc413b69fc2e0c994a91"} Oct 30 22:17:38.416992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.416951 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:38.417193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.417012 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:38.476237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.476138 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podStartSLOduration=5.476124793 podStartE2EDuration="5.476124793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:38.472325602 +0000 UTC m=+1386.441145167" watchObservedRunningTime="2026-10-30 22:17:38.476124793 +0000 UTC m=+1386.444944357" Oct 30 22:17:38.661072 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:38.661031 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b3da33_955a_45c9_862b_a6c70b8879cb.slice/crio-1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb WatchSource:0}: Error finding container 1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb: Status 404 returned error can't find the container with id 1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb Oct 30 22:17:38.661295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.661256 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" podStartSLOduration=3.661244012 podStartE2EDuration="3.661244012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:38.593511504 +0000 UTC m=+1386.562331069" watchObservedRunningTime="2026-10-30 22:17:38.661244012 +0000 UTC m=+1386.630063620" Oct 30 22:17:38.661732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:38.661714 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg"] Oct 30 22:17:39.339751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.339713 4292 generic.go:356] "Generic (PLEG): container finished" podID="29d6b4ca-3688-49db-acbc-02a111d043cc" containerID="d9b8177d7033d96a5b57334ee7c10f53aa2803110b85fc413b69fc2e0c994a91" exitCode=0 Oct 30 22:17:39.340213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.339801 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" event={"ID":"29d6b4ca-3688-49db-acbc-02a111d043cc","Type":"ContainerDied","Data":"d9b8177d7033d96a5b57334ee7c10f53aa2803110b85fc413b69fc2e0c994a91"} Oct 30 22:17:39.341766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.341743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" event={"ID":"e5b3da33-955a-45c9-862b-a6c70b8879cb","Type":"ContainerStarted","Data":"a1a691110c0304bd52208fc95d478455bde97fe56c7c9f10baefa6d4afd82d7e"} Oct 30 22:17:39.341905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.341782 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" event={"ID":"e5b3da33-955a-45c9-862b-a6c70b8879cb","Type":"ContainerStarted","Data":"1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb"} Oct 30 22:17:39.407843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.405937 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:39.407843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.406005 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:39.610665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:39.586124 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" podStartSLOduration=2.586103148 podStartE2EDuration="2.586103148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:39.584450175 +0000 UTC m=+1387.553269739" watchObservedRunningTime="2026-10-30 22:17:39.586103148 +0000 UTC m=+1387.554922713" Oct 30 22:17:40.352204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.352167 4292 generic.go:356] "Generic (PLEG): container finished" podID="e5b3da33-955a-45c9-862b-a6c70b8879cb" containerID="a1a691110c0304bd52208fc95d478455bde97fe56c7c9f10baefa6d4afd82d7e" exitCode=0 Oct 30 22:17:40.352649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.352252 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" event={"ID":"e5b3da33-955a-45c9-862b-a6c70b8879cb","Type":"ContainerDied","Data":"a1a691110c0304bd52208fc95d478455bde97fe56c7c9f10baefa6d4afd82d7e"} Oct 30 22:17:40.410601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.409925 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:40.410601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.409985 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:40.570596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.570559 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:40.661973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.661897 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\") pod \"29d6b4ca-3688-49db-acbc-02a111d043cc\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " Oct 30 22:17:40.662106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.662050 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\" (UniqueName: \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\") pod \"29d6b4ca-3688-49db-acbc-02a111d043cc\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " Oct 30 22:17:40.662106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.662075 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\") pod \"29d6b4ca-3688-49db-acbc-02a111d043cc\" (UID: \"29d6b4ca-3688-49db-acbc-02a111d043cc\") " Oct 30 22:17:40.662275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.662240 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca" pod "29d6b4ca-3688-49db-acbc-02a111d043cc" (UID: "29d6b4ca-3688-49db-acbc-02a111d043cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:40.662685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.662666 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:40.664217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.664191 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca" pod "29d6b4ca-3688-49db-acbc-02a111d043cc" (UID: "29d6b4ca-3688-49db-acbc-02a111d043cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:40.664304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.664201 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j" pod "29d6b4ca-3688-49db-acbc-02a111d043cc" (UID: "29d6b4ca-3688-49db-acbc-02a111d043cc"). InnerVolumeSpecName "kube-api-access-kct2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:40.764518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.764487 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kct2j\" (UniqueName: \"kubernetes.io/projected/29d6b4ca-3688-49db-acbc-02a111d043cc-kube-api-access-kct2j\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:40.764518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:40.764518 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d6b4ca-3688-49db-acbc-02a111d043cc-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:41.364404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.364370 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" Oct 30 22:17:41.364404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.364392 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt" event={"ID":"29d6b4ca-3688-49db-acbc-02a111d043cc","Type":"ContainerDied","Data":"fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1"} Oct 30 22:17:41.364971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.364457 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa80ed9d900001375545b26041e31f7e0533abfd8d3d6109d45c0cd553d62be1" Oct 30 22:17:41.413599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.408947 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:41.413599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.409009 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:41.583458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.583435 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:41.677109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.677024 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\" (UniqueName: \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\") pod \"e5b3da33-955a-45c9-862b-a6c70b8879cb\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " Oct 30 22:17:41.677109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.677064 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\") pod \"e5b3da33-955a-45c9-862b-a6c70b8879cb\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " Oct 30 22:17:41.677109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.677093 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\") pod \"e5b3da33-955a-45c9-862b-a6c70b8879cb\" (UID: \"e5b3da33-955a-45c9-862b-a6c70b8879cb\") " Oct 30 22:17:41.677544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.677520 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca" pod "e5b3da33-955a-45c9-862b-a6c70b8879cb" (UID: "e5b3da33-955a-45c9-862b-a6c70b8879cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:41.679129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.679098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca" pod "e5b3da33-955a-45c9-862b-a6c70b8879cb" (UID: "e5b3da33-955a-45c9-862b-a6c70b8879cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:41.679231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.679199 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt" pod "e5b3da33-955a-45c9-862b-a6c70b8879cb" (UID: "e5b3da33-955a-45c9-862b-a6c70b8879cb"). InnerVolumeSpecName "kube-api-access-592bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:41.778619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.778558 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592bt\" (UniqueName: \"kubernetes.io/projected/e5b3da33-955a-45c9-862b-a6c70b8879cb-kube-api-access-592bt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:41.778619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.778615 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:41.778619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:41.778626 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b3da33-955a-45c9-862b-a6c70b8879cb-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:42.374245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:42.374216 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" Oct 30 22:17:42.374641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:42.374240 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg" event={"ID":"e5b3da33-955a-45c9-862b-a6c70b8879cb","Type":"ContainerDied","Data":"1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb"} Oct 30 22:17:42.374641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:42.374292 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1515b5eb4a2a8fc895a87d1048f6d47ef4f6627402b6df79441062a5861e89fb" Oct 30 22:17:42.408116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:42.407955 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:42.408116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:42.408020 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:43.401738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:43.401701 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:17:43.402158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:43.401777 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:17:44.018421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.018385 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685"] Oct 30 22:17:44.019407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019381 4292 cpu_manager.go:414] "containerMap: removing container" podUID="29d6b4ca-3688-49db-acbc-02a111d043cc" containerName="revoke-keys" Oct 30 22:17:44.019407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019406 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d6b4ca-3688-49db-acbc-02a111d043cc" containerName="revoke-keys" Oct 30 22:17:44.019621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019471 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e5b3da33-955a-45c9-862b-a6c70b8879cb" containerName="revoke-keys" Oct 30 22:17:44.019621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019481 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b3da33-955a-45c9-862b-a6c70b8879cb" containerName="revoke-keys" Oct 30 22:17:44.019737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019662 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b3da33-955a-45c9-862b-a6c70b8879cb" containerName="revoke-keys" Oct 30 22:17:44.019737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.019691 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d6b4ca-3688-49db-acbc-02a111d043cc" containerName="revoke-keys" Oct 30 22:17:44.023980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.023953 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.068415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.068385 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2"] Oct 30 22:17:44.076283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.076247 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74908f82-rz9m2"] Oct 30 22:17:44.111318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.111059 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:44.111318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.111156 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:44.115332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.115306 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:44.115460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.115377 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:17:44.116909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.116884 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685"] Oct 30 22:17:44.130866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.130838 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.131030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.131012 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.131269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.131241 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkwg\" (UniqueName: \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.148900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.148874 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f5c000-894f-441f-a79d-2aed7a8bd8f6" path="/var/lib/kubelet/pods/82f5c000-894f-441f-a79d-2aed7a8bd8f6/volumes" Oct 30 22:17:44.232666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.232633 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.232803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.232793 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkwg\" (UniqueName: \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.232885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.232858 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.233489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.233465 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.235111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.235089 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.295946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.295918 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkwg\" (UniqueName: \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\") pod \"maas-api-revoke-sub-0ec14fab-gm685\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.339665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.339638 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:44.409876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.409847 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:44.410804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.410743 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:44.412142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.412086 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:17:44.728352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:44.728321 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685"] Oct 30 22:17:45.403836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:45.403796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" event={"ID":"2289fad4-dd54-4407-9385-ff14d5b19faf","Type":"ContainerStarted","Data":"1d01239ecff3298c08a48303b94bd203fa4ad34241a13e094ddf70cc3c5053b0"} Oct 30 22:17:45.404025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:45.403893 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" event={"ID":"2289fad4-dd54-4407-9385-ff14d5b19faf","Type":"ContainerStarted","Data":"56acaeabefdfd576fbe9f56f85a370d86ea420a2c6da50509d96a387aa41a269"} Oct 30 22:17:45.523543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:45.523493 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" podStartSLOduration=2.523479682 podStartE2EDuration="2.523479682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:45.522533455 +0000 UTC m=+1393.491353030" watchObservedRunningTime="2026-10-30 22:17:45.523479682 +0000 UTC m=+1393.492299324" Oct 30 22:17:46.414459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:46.414424 4292 generic.go:356] "Generic (PLEG): container finished" podID="2289fad4-dd54-4407-9385-ff14d5b19faf" containerID="1d01239ecff3298c08a48303b94bd203fa4ad34241a13e094ddf70cc3c5053b0" exitCode=0 Oct 30 22:17:46.414658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:46.414508 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" event={"ID":"2289fad4-dd54-4407-9385-ff14d5b19faf","Type":"ContainerDied","Data":"1d01239ecff3298c08a48303b94bd203fa4ad34241a13e094ddf70cc3c5053b0"} Oct 30 22:17:47.601169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.601140 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:47.677754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.677672 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\") pod \"2289fad4-dd54-4407-9385-ff14d5b19faf\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " Oct 30 22:17:47.677909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.677813 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\" (UniqueName: \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\") pod \"2289fad4-dd54-4407-9385-ff14d5b19faf\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " Oct 30 22:17:47.677909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.677862 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\") pod \"2289fad4-dd54-4407-9385-ff14d5b19faf\" (UID: \"2289fad4-dd54-4407-9385-ff14d5b19faf\") " Oct 30 22:17:47.678028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.677986 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca" pod "2289fad4-dd54-4407-9385-ff14d5b19faf" (UID: "2289fad4-dd54-4407-9385-ff14d5b19faf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:47.678538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.678521 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:47.680002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.679968 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca" pod "2289fad4-dd54-4407-9385-ff14d5b19faf" (UID: "2289fad4-dd54-4407-9385-ff14d5b19faf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:47.680002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.679979 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg" pod "2289fad4-dd54-4407-9385-ff14d5b19faf" (UID: "2289fad4-dd54-4407-9385-ff14d5b19faf"). InnerVolumeSpecName "kube-api-access-qxkwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:47.780211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.780180 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkwg\" (UniqueName: \"kubernetes.io/projected/2289fad4-dd54-4407-9385-ff14d5b19faf-kube-api-access-qxkwg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:47.780211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:47.780211 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2289fad4-dd54-4407-9385-ff14d5b19faf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:48.228287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.228248 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8"] Oct 30 22:17:48.238556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.238516 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2289fad4-dd54-4407-9385-ff14d5b19faf" containerName="revoke-keys" Oct 30 22:17:48.238556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.238555 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2289fad4-dd54-4407-9385-ff14d5b19faf" containerName="revoke-keys" Oct 30 22:17:48.241753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.239034 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2289fad4-dd54-4407-9385-ff14d5b19faf" containerName="revoke-keys" Oct 30 22:17:48.244157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.244138 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.290797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.290767 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8"] Oct 30 22:17:48.388591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.388546 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zxf\" (UniqueName: \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.388727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.388599 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.388789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.388754 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.433901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.433822 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" event={"ID":"2289fad4-dd54-4407-9385-ff14d5b19faf","Type":"ContainerDied","Data":"56acaeabefdfd576fbe9f56f85a370d86ea420a2c6da50509d96a387aa41a269"} Oct 30 22:17:48.433901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.433875 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56acaeabefdfd576fbe9f56f85a370d86ea420a2c6da50509d96a387aa41a269" Oct 30 22:17:48.433901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.433838 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685" Oct 30 22:17:48.489895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.489870 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.490050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.490002 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zxf\" (UniqueName: \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.490050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.490033 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.490728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.490705 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.492233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.492209 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.553097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.553065 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp"] Oct 30 22:17:48.556958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.556937 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zxf\" (UniqueName: \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\") pod \"maas-api-revoke-sub-21e32564-twvg8\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.557677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.557659 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.558121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.558101 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:48.623008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.622983 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp"] Oct 30 22:17:48.693508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.693425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.693714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.693690 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.693767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.693742 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59dk\" (UniqueName: \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.795226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.795196 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.795420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.795300 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.795420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.795327 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59dk\" (UniqueName: \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.796010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.795986 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.797554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.797534 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.858307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.858282 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59dk\" (UniqueName: \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\") pod \"maas-api-revoke-sub-68ae279e-k6jzp\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.874151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.874126 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:48.945894 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:48.943284 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ea30cc_3190_46c7_9c6b_d1eeb8054ba5.slice/crio-7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4 WatchSource:0}: Error finding container 7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4: Status 404 returned error can't find the container with id 7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4 Oct 30 22:17:48.957638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:48.945960 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8"] Oct 30 22:17:49.253831 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:49.253795 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdfaac1_06d2_4602_8967_a0b7c45c76fd.slice/crio-62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a WatchSource:0}: Error finding container 62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a: Status 404 returned error can't find the container with id 62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a Oct 30 22:17:49.255352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.255324 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp"] Oct 30 22:17:49.444899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.444864 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" event={"ID":"ebdfaac1-06d2-4602-8967-a0b7c45c76fd","Type":"ContainerStarted","Data":"62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a"} Oct 30 22:17:49.446849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.446813 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" event={"ID":"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5","Type":"ContainerStarted","Data":"1594204f4d71ef1bdab28873a5796e98ff9975b8047494e3f2e8da0097b8d868"} Oct 30 22:17:49.446997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.446859 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" event={"ID":"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5","Type":"ContainerStarted","Data":"7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4"} Oct 30 22:17:49.516824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.516781 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:17:49.521499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.521474 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.572730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.572672 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" podStartSLOduration=1.572656931 podStartE2EDuration="1.572656931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:49.567450989 +0000 UTC m=+1397.536270553" watchObservedRunningTime="2026-10-30 22:17:49.572656931 +0000 UTC m=+1397.541476494" Oct 30 22:17:49.615045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.614981 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615039 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615147 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615191 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615334 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46rs\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615366 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615439 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615513 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.615642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.615613 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.617064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.617038 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:17:49.717342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717315 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717356 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717380 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717399 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717486 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46rs\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717506 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717567 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717674 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717728 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717874 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.717923 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.718000 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.718177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.718562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.718444 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.719676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.719653 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.719887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.719869 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.780111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.780045 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.781089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.781070 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46rs\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\") pod \"e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:49.836963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:49.836940 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:50.270824 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:50.270790 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d1959d_c544_483c_a7fc_0e0ec67f3629.slice/crio-fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce WatchSource:0}: Error finding container fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce: Status 404 returned error can't find the container with id fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce Oct 30 22:17:50.271626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.271561 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:17:50.273206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.273173 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:50.273280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.273237 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:50.273280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.273267 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:50.461316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.461271 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" event={"ID":"68d1959d-c544-483c-a7fc-0e0ec67f3629","Type":"ContainerStarted","Data":"0a30f6576a42f934140736cf217af57b602322bba70c7add156b0c20f4092e58"} Oct 30 22:17:50.461470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.461345 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" event={"ID":"68d1959d-c544-483c-a7fc-0e0ec67f3629","Type":"ContainerStarted","Data":"fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce"} Oct 30 22:17:50.464188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.464153 4292 generic.go:356] "Generic (PLEG): container finished" podID="ebdfaac1-06d2-4602-8967-a0b7c45c76fd" containerID="e1ccd44a7fdd264f2ec700febf45e0b3d18795fe5c57e0d75f879d6bf8ea3e3e" exitCode=0 Oct 30 22:17:50.464398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.464375 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" event={"ID":"ebdfaac1-06d2-4602-8967-a0b7c45c76fd","Type":"ContainerDied","Data":"e1ccd44a7fdd264f2ec700febf45e0b3d18795fe5c57e0d75f879d6bf8ea3e3e"} Oct 30 22:17:50.466010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.465982 4292 generic.go:356] "Generic (PLEG): container finished" podID="61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" containerID="1594204f4d71ef1bdab28873a5796e98ff9975b8047494e3f2e8da0097b8d868" exitCode=0 Oct 30 22:17:50.466139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.466124 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" event={"ID":"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5","Type":"ContainerDied","Data":"1594204f4d71ef1bdab28873a5796e98ff9975b8047494e3f2e8da0097b8d868"} Oct 30 22:17:50.573956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.573924 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:17:50.578745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.578722 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.597216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.597160 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podStartSLOduration=1.5971410480000001 podStartE2EDuration="1.597141048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:50.590327321 +0000 UTC m=+1398.559146885" watchObservedRunningTime="2026-10-30 22:17:50.597141048 +0000 UTC m=+1398.565960612" Oct 30 22:17:50.643929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.643899 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:17:50.733413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733382 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.733784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmbz\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.733784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733523 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.733784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733756 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.733904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733791 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.733904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733884 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.734012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.733979 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.734137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.734072 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.734137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.734109 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835515 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835557 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835623 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835679 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835726 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835750 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.835954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835858 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.835979 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmbz\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.836108 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.836220 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.836372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.837264 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:50.837692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.837438 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.837519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.837806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.837787 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.839079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.839053 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.839199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.839125 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.902398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.900983 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmbz\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.902398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.901862 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\") pod \"e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:50.902398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:50.902134 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:51.361211 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:51.361183 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcacb8fd_a8b2_4a1f_8ef1_c72d2043f267.slice/crio-0695068ec6a81884cd234b85c0c86e5614a13bb62aedbbe06b8eb00218cf08c9 WatchSource:0}: Error finding container 0695068ec6a81884cd234b85c0c86e5614a13bb62aedbbe06b8eb00218cf08c9: Status 404 returned error can't find the container with id 0695068ec6a81884cd234b85c0c86e5614a13bb62aedbbe06b8eb00218cf08c9 Oct 30 22:17:51.361938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.361912 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:17:51.364103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.364074 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:51.364167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.364136 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:51.364199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.364167 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:51.478598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.478537 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" event={"ID":"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267","Type":"ContainerStarted","Data":"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc"} Oct 30 22:17:51.478715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.478649 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" event={"ID":"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267","Type":"ContainerStarted","Data":"0695068ec6a81884cd234b85c0c86e5614a13bb62aedbbe06b8eb00218cf08c9"} Oct 30 22:17:51.642068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.642019 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podStartSLOduration=1.642003968 podStartE2EDuration="1.642003968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:51.641324384 +0000 UTC m=+1399.610143949" watchObservedRunningTime="2026-10-30 22:17:51.642003968 +0000 UTC m=+1399.610823532" Oct 30 22:17:51.757974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.754493 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:51.831652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.831042 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:51.839364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.839339 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:51.839475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.839385 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:51.857685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.857657 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\") pod \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " Oct 30 22:17:51.857930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.857911 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\") pod \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " Oct 30 22:17:51.858005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.857991 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\" (UniqueName: \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\") pod \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\" (UID: \"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5\") " Oct 30 22:17:51.858253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.858229 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca" pod "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" (UID: "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:51.858600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.858563 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.859949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.859926 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca" pod "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" (UID: "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:51.860194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.860169 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf" pod "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" (UID: "61ea30cc-3190-46c7-9c6b-d1eeb8054ba5"). InnerVolumeSpecName "kube-api-access-74zxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:51.913213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.913126 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:51.967600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.965809 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\") pod \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " Oct 30 22:17:51.967600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.966017 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\" (UniqueName: \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\") pod \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " Oct 30 22:17:51.967600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.966103 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\") pod \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\" (UID: \"ebdfaac1-06d2-4602-8967-a0b7c45c76fd\") " Oct 30 22:17:51.967600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.966761 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.967600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.966785 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zxf\" (UniqueName: \"kubernetes.io/projected/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5-kube-api-access-74zxf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.985627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:51.969773 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca" pod "ebdfaac1-06d2-4602-8967-a0b7c45c76fd" (UID: "ebdfaac1-06d2-4602-8967-a0b7c45c76fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:52.060599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.056732 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca" pod "ebdfaac1-06d2-4602-8967-a0b7c45c76fd" (UID: "ebdfaac1-06d2-4602-8967-a0b7c45c76fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:52.070187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.070151 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.070187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.070186 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.070404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.070212 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg"] Oct 30 22:17:52.075784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.075751 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk" pod "ebdfaac1-06d2-4602-8967-a0b7c45c76fd" (UID: "ebdfaac1-06d2-4602-8967-a0b7c45c76fd"). InnerVolumeSpecName "kube-api-access-x59dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:52.077612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.077557 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7d6bf6-kxktg"] Oct 30 22:17:52.151133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.150872 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8179c7f-678d-464a-9c0b-8143c3f33d35" path="/var/lib/kubelet/pods/a8179c7f-678d-464a-9c0b-8143c3f33d35/volumes" Oct 30 22:17:52.224312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.224276 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59dk\" (UniqueName: \"kubernetes.io/projected/ebdfaac1-06d2-4602-8967-a0b7c45c76fd-kube-api-access-x59dk\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.468443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.468364 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:17:52.469048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469027 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ebdfaac1-06d2-4602-8967-a0b7c45c76fd" containerName="revoke-keys" Oct 30 22:17:52.469174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469162 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdfaac1-06d2-4602-8967-a0b7c45c76fd" containerName="revoke-keys" Oct 30 22:17:52.469229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469222 4292 cpu_manager.go:414] "containerMap: removing container" podUID="61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" containerName="revoke-keys" Oct 30 22:17:52.469273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469266 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" containerName="revoke-keys" Oct 30 22:17:52.469512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469504 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdfaac1-06d2-4602-8967-a0b7c45c76fd" containerName="revoke-keys" Oct 30 22:17:52.469564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.469558 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" containerName="revoke-keys" Oct 30 22:17:52.474600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.474584 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.509856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.508874 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" event={"ID":"ebdfaac1-06d2-4602-8967-a0b7c45c76fd","Type":"ContainerDied","Data":"62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a"} Oct 30 22:17:52.509856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.508940 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62039299dcd5d61f522cb8f9ea5933cf8e8ed1e87a395ab6ee865413b1ec459a" Oct 30 22:17:52.509856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.509038 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.556877 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.556938 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.556963 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557001 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5tr\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557133 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557150 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557189 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.557966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.557217 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.561418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.558958 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" Oct 30 22:17:52.561418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.558969 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8" event={"ID":"61ea30cc-3190-46c7-9c6b-d1eeb8054ba5","Type":"ContainerDied","Data":"7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4"} Oct 30 22:17:52.561418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.559006 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aec98db366d9585d184a7ee159dc733799310cb53a8ef491bc5c12f26f13ac4" Oct 30 22:17:52.566005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.565981 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:17:52.605992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.605963 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-73fb6577-openshift-default-dockercfg-gd8pj\"" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.659699 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.659817 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.659848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.659909 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.660018 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5tr\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.660181 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.660208 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.660296 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.660374 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.661442 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.662051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.661709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.663049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.663004 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.663466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.663419 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.663828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.663805 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.664195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.664177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.666156 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.666132 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.746654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.731999 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.746654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.737185 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb"] Oct 30 22:17:52.759660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.758605 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.760859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.760811 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5tr\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\") pod \"e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.765302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.765127 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb"] Oct 30 22:17:52.816974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.807994 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:52.836651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.821442 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:52.836651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.821550 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:52.836651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.821622 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:52.836651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.821721 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:17:52.855272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.838979 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:52.855272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.839048 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:52.863880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.863828 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5ls\" (UniqueName: \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.864107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.864044 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.867621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.866911 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.950597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.949877 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:52.950597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.949933 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:52.968133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.968101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5ls\" (UniqueName: \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.968286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.968226 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.968326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.968292 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.969118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.969091 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:52.970630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:52.970611 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:53.036724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.036389 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5ls\" (UniqueName: \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\") pod \"maas-api-revoke-sub-4c333aa4-57nrb\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:53.162710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.162675 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:53.376914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.374966 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:17:53.384128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.384084 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:53.384244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.384178 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:53.384244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.384225 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:53.572326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.572296 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerStarted","Data":"b5303008dfb2356648c6ccf907c3615108c041518c6602e7effc231c141052f8"} Oct 30 22:17:53.652447 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:53.652419 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73415ce_d3fc_43b0_a966_43007a4af66d.slice/crio-d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6 WatchSource:0}: Error finding container d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6: Status 404 returned error can't find the container with id d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6 Oct 30 22:17:53.654065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.652774 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb"] Oct 30 22:17:53.838985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.838889 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:53.839506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.839477 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:53.903958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.903925 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:53.904208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:53.904181 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:54.581666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.581627 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" event={"ID":"f73415ce-d3fc-43b0-a966-43007a4af66d","Type":"ContainerStarted","Data":"8a7bfdc98cbeba4e3f6d962262afb184e9b4e678e7ed6968f77ea8142af4d581"} Oct 30 22:17:54.581931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.581909 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" event={"ID":"f73415ce-d3fc-43b0-a966-43007a4af66d","Type":"ContainerStarted","Data":"d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6"} Oct 30 22:17:54.583255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.583204 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerStarted","Data":"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a"} Oct 30 22:17:54.737852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.732631 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" podStartSLOduration=2.732612033 podStartE2EDuration="2.732612033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:54.70438112 +0000 UTC m=+1402.673200687" watchObservedRunningTime="2026-10-30 22:17:54.732612033 +0000 UTC m=+1402.701431598" Oct 30 22:17:54.822914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.822844 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:54.843676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.840478 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:54.843676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.841334 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:54.843676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.841686 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:54.846563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.845089 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podStartSLOduration=2.845067856 podStartE2EDuration="2.845067856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:54.831194976 +0000 UTC m=+1402.800014541" watchObservedRunningTime="2026-10-30 22:17:54.845067856 +0000 UTC m=+1402.813887422" Oct 30 22:17:54.913596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.906971 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:54.913596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.907043 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:54.942959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.942921 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489"] Oct 30 22:17:54.947432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:54.947412 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.077426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.077384 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.077859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.077830 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67b4\" (UniqueName: \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.078150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.078132 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.080250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.080037 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489"] Oct 30 22:17:55.179512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.179388 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.179694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.179521 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.179764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.179701 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67b4\" (UniqueName: \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.180184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.180157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.181938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.181916 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.243451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.243416 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67b4\" (UniqueName: \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\") pod \"maas-api-revoke-sub-8b5f5b6b-md489\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.264956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.264927 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:55.669913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.669855 4292 generic.go:356] "Generic (PLEG): container finished" podID="f73415ce-d3fc-43b0-a966-43007a4af66d" containerID="8a7bfdc98cbeba4e3f6d962262afb184e9b4e678e7ed6968f77ea8142af4d581" exitCode=0 Oct 30 22:17:55.670298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.670259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" event={"ID":"f73415ce-d3fc-43b0-a966-43007a4af66d","Type":"ContainerDied","Data":"8a7bfdc98cbeba4e3f6d962262afb184e9b4e678e7ed6968f77ea8142af4d581"} Oct 30 22:17:55.671163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.671130 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:55.672743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.672721 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:17:55.744284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.744255 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489"] Oct 30 22:17:55.746413 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:17:55.746379 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0fd961_9ba2_4869_8239_62a5bbc58cc7.slice/crio-e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f WatchSource:0}: Error finding container e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f: Status 404 returned error can't find the container with id e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f Oct 30 22:17:55.841748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.840737 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:55.841748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.840782 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:55.921663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.919135 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:55.921663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:55.919194 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:56.640693 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:17:56.640657 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:17:56.682818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.682784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" event={"ID":"ea0fd961-9ba2-4869-8239-62a5bbc58cc7","Type":"ContainerStarted","Data":"7a4e661b98fe0e4e72a3116c18d9507e7f5e302eb463c7de2b42073dcdcb61c7"} Oct 30 22:17:56.682990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.682838 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" event={"ID":"ea0fd961-9ba2-4869-8239-62a5bbc58cc7","Type":"ContainerStarted","Data":"e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f"} Oct 30 22:17:56.838637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.835344 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" podStartSLOduration=2.835327185 podStartE2EDuration="2.835327185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:56.825232344 +0000 UTC m=+1404.794051908" watchObservedRunningTime="2026-10-30 22:17:56.835327185 +0000 UTC m=+1404.804146748" Oct 30 22:17:56.839742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.839715 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 22:17:56.839791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.839769 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 22:17:56.917658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.906544 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:56.917658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.906614 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:56.940461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.940434 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:56.984166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.982813 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\") pod \"f73415ce-d3fc-43b0-a966-43007a4af66d\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " Oct 30 22:17:56.984166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.983013 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\" (UniqueName: \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\") pod \"f73415ce-d3fc-43b0-a966-43007a4af66d\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " Oct 30 22:17:56.984166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.983157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\") pod \"f73415ce-d3fc-43b0-a966-43007a4af66d\" (UID: \"f73415ce-d3fc-43b0-a966-43007a4af66d\") " Oct 30 22:17:56.984487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:56.984465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca" pod "f73415ce-d3fc-43b0-a966-43007a4af66d" (UID: "f73415ce-d3fc-43b0-a966-43007a4af66d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:57.019732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.019690 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca" pod "f73415ce-d3fc-43b0-a966-43007a4af66d" (UID: "f73415ce-d3fc-43b0-a966-43007a4af66d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:57.019732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.019712 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls" pod "f73415ce-d3fc-43b0-a966-43007a4af66d" (UID: "f73415ce-d3fc-43b0-a966-43007a4af66d"). InnerVolumeSpecName "kube-api-access-ng5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:57.091617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.085449 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:57.091835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.091811 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73415ce-d3fc-43b0-a966-43007a4af66d-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:57.091964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.091947 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5ls\" (UniqueName: \"kubernetes.io/projected/f73415ce-d3fc-43b0-a966-43007a4af66d-kube-api-access-ng5ls\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:57.716926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.716882 4292 generic.go:356] "Generic (PLEG): container finished" podID="ea0fd961-9ba2-4869-8239-62a5bbc58cc7" containerID="7a4e661b98fe0e4e72a3116c18d9507e7f5e302eb463c7de2b42073dcdcb61c7" exitCode=0 Oct 30 22:17:57.717151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.716974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" event={"ID":"ea0fd961-9ba2-4869-8239-62a5bbc58cc7","Type":"ContainerDied","Data":"7a4e661b98fe0e4e72a3116c18d9507e7f5e302eb463c7de2b42073dcdcb61c7"} Oct 30 22:17:57.718863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.718839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" Oct 30 22:17:57.719025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.718879 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb" event={"ID":"f73415ce-d3fc-43b0-a966-43007a4af66d","Type":"ContainerDied","Data":"d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6"} Oct 30 22:17:57.719151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.719137 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d465c973dc541d1577cc9ad4a95adb40eac00391cce6f9cfd9f4ea042d6dedb6" Oct 30 22:17:57.906599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.905449 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:57.906599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:57.905520 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:58.023324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.023238 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:58.023749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.023707 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:58.024603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.024551 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:17:58.903063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.903033 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:17:58.903174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.903088 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:17:58.906279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:58.906258 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:59.039082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.039001 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\") pod \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " Oct 30 22:17:59.039082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.039075 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\") pod \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " Oct 30 22:17:59.039554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.039320 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\" (UniqueName: \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\") pod \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\" (UID: \"ea0fd961-9ba2-4869-8239-62a5bbc58cc7\") " Oct 30 22:17:59.040677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.040648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca" pod "ea0fd961-9ba2-4869-8239-62a5bbc58cc7" (UID: "ea0fd961-9ba2-4869-8239-62a5bbc58cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:59.042932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.042901 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca" pod "ea0fd961-9ba2-4869-8239-62a5bbc58cc7" (UID: "ea0fd961-9ba2-4869-8239-62a5bbc58cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:59.043883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.043857 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4" pod "ea0fd961-9ba2-4869-8239-62a5bbc58cc7" (UID: "ea0fd961-9ba2-4869-8239-62a5bbc58cc7"). InnerVolumeSpecName "kube-api-access-m67b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:59.141129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.141091 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:59.141129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.141118 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:59.141129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.141127 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m67b4\" (UniqueName: \"kubernetes.io/projected/ea0fd961-9ba2-4869-8239-62a5bbc58cc7-kube-api-access-m67b4\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:59.398793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.398759 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr"] Oct 30 22:17:59.399437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399417 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ea0fd961-9ba2-4869-8239-62a5bbc58cc7" containerName="revoke-keys" Oct 30 22:17:59.399437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399434 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0fd961-9ba2-4869-8239-62a5bbc58cc7" containerName="revoke-keys" Oct 30 22:17:59.399652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399448 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f73415ce-d3fc-43b0-a966-43007a4af66d" containerName="revoke-keys" Oct 30 22:17:59.399652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399454 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73415ce-d3fc-43b0-a966-43007a4af66d" containerName="revoke-keys" Oct 30 22:17:59.399652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399599 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73415ce-d3fc-43b0-a966-43007a4af66d" containerName="revoke-keys" Oct 30 22:17:59.399652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.399620 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0fd961-9ba2-4869-8239-62a5bbc58cc7" containerName="revoke-keys" Oct 30 22:17:59.402802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.402778 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.473254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.473224 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr"] Oct 30 22:17:59.547146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.547101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnftj\" (UniqueName: \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.547315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.547179 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.547315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.547299 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.648526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.648492 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.648694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.648544 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.648779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.648762 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnftj\" (UniqueName: \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.649366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.649323 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.650873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.650851 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.711744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.711718 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnftj\" (UniqueName: \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\") pod \"maas-api-revoke-sub-100d9e93-7wpwr\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.739101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.739073 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" Oct 30 22:17:59.739249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.739101 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489" event={"ID":"ea0fd961-9ba2-4869-8239-62a5bbc58cc7","Type":"ContainerDied","Data":"e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f"} Oct 30 22:17:59.739249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.739148 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e878845ffac0584a8da37c5cac4ab7b8dce3113be58ce95d0f1e84c1ebca7b9f" Oct 30 22:17:59.743476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.743456 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:17:59.907627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.907549 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:59.908060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.908035 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:17:59.908682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:17:59.908661 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:18:00.148938 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:00.148905 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656d48c0_69db_4eba_a11f_267f4344aedf.slice/crio-427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf WatchSource:0}: Error finding container 427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf: Status 404 returned error can't find the container with id 427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf Oct 30 22:18:00.151892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:00.151867 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr"] Oct 30 22:18:00.750633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:00.750526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" event={"ID":"656d48c0-69db-4eba-a11f-267f4344aedf","Type":"ContainerStarted","Data":"d11a6314cd1cfaace331e8482441353f62d7edf3cfcdf773e0c3c7b167ad8b80"} Oct 30 22:18:00.750633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:00.750624 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" event={"ID":"656d48c0-69db-4eba-a11f-267f4344aedf","Type":"ContainerStarted","Data":"427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf"} Oct 30 22:18:00.870867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:00.870813 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" podStartSLOduration=1.870798852 podStartE2EDuration="1.870798852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:00.870309272 +0000 UTC m=+1408.839128835" watchObservedRunningTime="2026-10-30 22:18:00.870798852 +0000 UTC m=+1408.839618419" Oct 30 22:18:01.224227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.224195 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:18:01.228525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.228501 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.286712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.286678 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ea4ac3ec-openshift-default-dockercfg-lpxhz\"" Oct 30 22:18:01.293943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.293913 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:18:01.382826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.382734 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.382966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.382901 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6xk\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383008 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383057 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383109 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383157 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383298 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.383394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.383351 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485089 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6xk\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485189 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485270 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485494 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.485552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.485533 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.486452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.486421 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.488416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.488395 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.490823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.490796 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.491039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.490982 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.491182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.491155 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.491344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.491317 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.491493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.491415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.494851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.494826 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.561622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.561555 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.562073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.562046 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6xk\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\") pod \"e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:01.761007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.760921 4292 generic.go:356] "Generic (PLEG): container finished" podID="656d48c0-69db-4eba-a11f-267f4344aedf" containerID="d11a6314cd1cfaace331e8482441353f62d7edf3cfcdf773e0c3c7b167ad8b80" exitCode=0 Oct 30 22:18:01.761007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.760954 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" event={"ID":"656d48c0-69db-4eba-a11f-267f4344aedf","Type":"ContainerDied","Data":"d11a6314cd1cfaace331e8482441353f62d7edf3cfcdf773e0c3c7b167ad8b80"} Oct 30 22:18:01.843662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:01.843626 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:02.312381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.312347 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:18:02.340754 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:02.340711 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb2bfca_4232_4aff_a804_a9cbd4d48e09.slice/crio-372b18ca32ec96142077857d1b9f2eaace2ee6a439d63320fc71a34e75392ba7 WatchSource:0}: Error finding container 372b18ca32ec96142077857d1b9f2eaace2ee6a439d63320fc71a34e75392ba7: Status 404 returned error can't find the container with id 372b18ca32ec96142077857d1b9f2eaace2ee6a439d63320fc71a34e75392ba7 Oct 30 22:18:02.343257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.343222 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:02.343361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.343312 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:02.343421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.343361 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:02.776282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.776194 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerStarted","Data":"0faa4ecfb13d8995fc1fca2164be136c2575747e499b3a725b3b39f03f3da0f1"} Oct 30 22:18:02.776486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.776284 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerStarted","Data":"372b18ca32ec96142077857d1b9f2eaace2ee6a439d63320fc71a34e75392ba7"} Oct 30 22:18:02.843864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.843829 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:02.976478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:02.976454 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:18:03.033244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.033142 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\") pod \"656d48c0-69db-4eba-a11f-267f4344aedf\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " Oct 30 22:18:03.033421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.033253 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\" (UniqueName: \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\") pod \"656d48c0-69db-4eba-a11f-267f4344aedf\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " Oct 30 22:18:03.033421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.033298 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\") pod \"656d48c0-69db-4eba-a11f-267f4344aedf\" (UID: \"656d48c0-69db-4eba-a11f-267f4344aedf\") " Oct 30 22:18:03.033923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.033893 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca" pod "656d48c0-69db-4eba-a11f-267f4344aedf" (UID: "656d48c0-69db-4eba-a11f-267f4344aedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:03.034110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.034089 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:03.035950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.035924 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj" pod "656d48c0-69db-4eba-a11f-267f4344aedf" (UID: "656d48c0-69db-4eba-a11f-267f4344aedf"). InnerVolumeSpecName "kube-api-access-xnftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:03.036592 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.036556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca" pod "656d48c0-69db-4eba-a11f-267f4344aedf" (UID: "656d48c0-69db-4eba-a11f-267f4344aedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:03.105637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.100378 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podStartSLOduration=2.100360336 podStartE2EDuration="2.100360336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:02.910209786 +0000 UTC m=+1410.879029350" watchObservedRunningTime="2026-10-30 22:18:03.100360336 +0000 UTC m=+1411.069179900" Oct 30 22:18:03.142635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.142595 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656d48c0-69db-4eba-a11f-267f4344aedf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:03.142635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.142638 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnftj\" (UniqueName: \"kubernetes.io/projected/656d48c0-69db-4eba-a11f-267f4344aedf-kube-api-access-xnftj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:03.784286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.784261 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" Oct 30 22:18:03.784286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.784265 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr" event={"ID":"656d48c0-69db-4eba-a11f-267f4344aedf","Type":"ContainerDied","Data":"427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf"} Oct 30 22:18:03.784702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.784315 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427f9e30bad8f043540c231b1311f006c222fbe92c795ae67ae5d40c5af8afcf" Oct 30 22:18:03.849913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:03.849886 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:04.792268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:04.792237 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:04.793315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:04.793293 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:18:05.415943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.415910 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:18:05.416769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.416748 4292 cpu_manager.go:414] "containerMap: removing container" podUID="656d48c0-69db-4eba-a11f-267f4344aedf" containerName="revoke-keys" Oct 30 22:18:05.416769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.416767 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656d48c0-69db-4eba-a11f-267f4344aedf" containerName="revoke-keys" Oct 30 22:18:05.416990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.416906 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="656d48c0-69db-4eba-a11f-267f4344aedf" containerName="revoke-keys" Oct 30 22:18:05.421026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.421005 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471027 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471201 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471261 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471283 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471302 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471474 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471498 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471537 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tlh\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.471694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.471666 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.510597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.510541 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:18:05.573599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573551 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573702 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573736 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573759 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573790 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573886 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.573970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573952 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.573983 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574025 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tlh\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574055 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574061 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574199 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574263 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.574733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.574709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.576250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.576227 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.576628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.576565 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.637431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.637401 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.637590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.637540 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tlh\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:05.738165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:05.738096 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:06.200896 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:06.200862 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c0e3c1_d663_474b_9e22_f0e1183abd52.slice/crio-c01ab042ba5a28e757be6ddaa2115744ecb777ffa141d77d52edaa80ec677672 WatchSource:0}: Error finding container c01ab042ba5a28e757be6ddaa2115744ecb777ffa141d77d52edaa80ec677672: Status 404 returned error can't find the container with id c01ab042ba5a28e757be6ddaa2115744ecb777ffa141d77d52edaa80ec677672 Oct 30 22:18:06.202673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.202533 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:18:06.203448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.203416 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:06.203515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.203477 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:06.203515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.203506 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:06.842254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.842215 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" event={"ID":"a6c0e3c1-d663-474b-9e22-f0e1183abd52","Type":"ContainerStarted","Data":"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb"} Oct 30 22:18:06.842506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.842281 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" event={"ID":"a6c0e3c1-d663-474b-9e22-f0e1183abd52","Type":"ContainerStarted","Data":"c01ab042ba5a28e757be6ddaa2115744ecb777ffa141d77d52edaa80ec677672"} Oct 30 22:18:06.972566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:06.972507 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podStartSLOduration=1.972493353 podStartE2EDuration="1.972493353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:06.967612909 +0000 UTC m=+1414.936432474" watchObservedRunningTime="2026-10-30 22:18:06.972493353 +0000 UTC m=+1414.941312916" Oct 30 22:18:07.738528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:07.738492 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:07.740018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:07.739991 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:07.740140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:07.740039 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:08.738791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:08.738754 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:08.739193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:08.738817 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:09.747597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:09.747510 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:09.748129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:09.747592 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:10.753605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:10.748655 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:10.753605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:10.748724 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:11.077911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.077514 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:11.122713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.122659 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.278483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.278432 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.295599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.279910 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.295599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.285653 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.387656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.377672 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.463703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.460146 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:18:11.463703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.460554 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:18:11.609282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.491504 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:11.609282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.492971 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:18:11.609282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.493271 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:18:11.609282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.493522 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 22:18:11.801683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.783209 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:11.801683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:11.783290 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:12.147665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.123235 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:18:12.149283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.125667 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:12.264637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.258116 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:18:12.264637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.258190 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:18:12.312616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.302147 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:12.312616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.303399 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:18:12.312616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.303717 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:18:12.312616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.303798 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:18:12.412174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.412069 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:12.413621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.413595 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:18:12.500666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.472876 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:18:12.500666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.498750 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:18:12.501466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.501428 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:12.502959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.502927 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:18:12.513787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.513750 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:12.515035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.515002 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:18:12.530733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.530692 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:18:12.579612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.558747 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:18:12.712477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.576699 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:18:12.720734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.720705 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:18:12.732951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.732916 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:18:12.736730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.736705 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:18:12.770286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.770254 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:12.770675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.770546 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:12.951262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:12.951225 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:18:13.152809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.152746 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:18:13.174601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.163639 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:13.205618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.200445 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.328989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.290811 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:18:13.341714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.331776 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:18:13.385938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.332362 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:18:13.386509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.347836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skrr\" (UniqueName: \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.386824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.386800 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.410411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.410331 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-73fb6577-dockercfg-8t2pl\"" Oct 30 22:18:13.424404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.411993 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-73fb6577\"" Oct 30 22:18:13.440308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.426065 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.488601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.484794 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.526924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.504529 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.527826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.506024 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skrr\" (UniqueName: \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.527826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.527607 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.539940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.539903 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:13.543652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.539761 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.566348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.519912 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 22:18:13.603599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.555472 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.603599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.563049 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:13.718652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.714092 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:18:13.871316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.871278 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:13.871616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.871565 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:13.927184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:13.927149 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skrr\" (UniqueName: \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\") pod \"payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:14.104702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.104661 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:14.130603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.125655 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.226600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.140851 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:14.240157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.169661 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.240703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.197956 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:14.393657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.389652 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:18:14.393657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.390056 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:18:14.436609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.420880 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xhqql" podUID="dcdab93d-64bb-41e6-8be7-f7bae6825a5c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.621731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.621677 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:18:14.631437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.622316 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.711923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.711822 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.716649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.661671 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 22:18:14.741845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.741804 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.742388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.740223 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:14.742585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.742547 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:14.776634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.747629 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:18:14.890170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.770772 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 22:18:14.890439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.841561 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 22:18:14.958635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.938041 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:14.958635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:14.938098 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:15.121200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.121150 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.153847 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:18:15.153796 4292 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.012s" Oct 30 22:18:15.194274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.194226 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:15.194546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.194500 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.245784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.245736 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.283450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.283408 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:15.396829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.396722 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:18:15.398754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.398718 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.622964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.622903 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.747006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.746889 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.748961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.748925 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:18:15.775136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.775077 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:15.834598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.815468 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:15.835594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.835533 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:15.939649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.878720 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:18:15.955167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.902755 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 22:18:15.959678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:15.903230 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:16.028785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.028683 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:16.029079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.029045 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:16.307598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.299844 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:16.308503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.300260 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:18:16.517403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.517337 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:18:16.600042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.599935 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:18:16.640842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.640799 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:16.818333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.818269 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:18:17.218951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.218901 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 22:18:17.306596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.305666 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 22:18:17.306596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.821279 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:18:17.321761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.321718 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:18:17.604072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.604012 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:17.604427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.886928 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:17.604512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.604458 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:17.634602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.628907 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:18:17.656649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.651602 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:18:17.656649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.652863 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 22:18:17.690157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.688801 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:18:17.690157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:16.821340 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 22:18:17.827639 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:18:17.752515 4292 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.595s" Oct 30 22:18:17.874379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.853962 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:17.967595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.934226 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:18.076900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.076861 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:18.077111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.008753 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:18.077290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:17.964103 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:18.139466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.139382 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:18:18.224882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.224692 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:18.352020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.346316 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:18:18.352020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.346382 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:18:18.352020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.346408 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:18:18.352020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.346458 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:18:18.352020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.346475 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp"] Oct 30 22:18:18.354091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.353963 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:18:18.354091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354009 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:18:18.354091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354041 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:18:18.354091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354075 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:18:18.354295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354110 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:18:18.354295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354142 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:18:18.355770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.354691 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.355770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.355311 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:18:18.367706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.367684 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp"] Oct 30 22:18:18.374154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.374022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" event={"ID":"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d","Type":"ContainerStarted","Data":"aea3c5a3b1896803a432549d86942cc6b06ddbe2321bc4b0e6bc4c9c5933b255"} Oct 30 22:18:18.424181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.423828 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:18.424181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.423948 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:18.451885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.451843 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.452058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.451957 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t49g\" (UniqueName: \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.453073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.453042 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.464119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.464083 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:18.464314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.464203 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:18:18.554959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.554916 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.555898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.555873 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.556002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.555955 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t49g\" (UniqueName: \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.556749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.556723 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.558496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.558421 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.622023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.621982 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t49g\" (UniqueName: \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\") pod \"maas-api-revoke-sub-5eeb8576-jlwjp\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.684339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.684271 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:18.741145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.740945 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:18:18.741145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:18.741018 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 22:18:19.095054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:19.095022 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp"] Oct 30 22:18:19.096803 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:19.096773 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc15dd84_b0d7_4190_93fe_995ece656ee3.slice/crio-f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c WatchSource:0}: Error finding container f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c: Status 404 returned error can't find the container with id f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c Oct 30 22:18:19.386197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:19.385757 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" event={"ID":"fc15dd84-b0d7-4190-93fe-995ece656ee3","Type":"ContainerStarted","Data":"f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c"} Oct 30 22:18:19.388400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:19.388343 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" event={"ID":"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d","Type":"ContainerStarted","Data":"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d"} Oct 30 22:18:19.388570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:19.388494 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:19.916969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:19.916913 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" podStartSLOduration=7.9168965369999995 podStartE2EDuration="7.916896537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:19.915680482 +0000 UTC m=+1427.884500043" watchObservedRunningTime="2026-10-30 22:18:19.916896537 +0000 UTC m=+1427.885716112" Oct 30 22:18:20.045553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:20.045515 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:20.045998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:20.045975 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:20.046749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:20.046731 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:18:20.401124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:20.401092 4292 generic.go:356] "Generic (PLEG): container finished" podID="fc15dd84-b0d7-4190-93fe-995ece656ee3" containerID="53f9406835410d30b5ced3b8c36b3e465ba6a3f65c1b2eb3ad742ff488b0e414" exitCode=0 Oct 30 22:18:20.401621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:20.401178 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" event={"ID":"fc15dd84-b0d7-4190-93fe-995ece656ee3","Type":"ContainerDied","Data":"53f9406835410d30b5ced3b8c36b3e465ba6a3f65c1b2eb3ad742ff488b0e414"} Oct 30 22:18:21.578038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.578013 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:21.712337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.712255 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\") pod \"fc15dd84-b0d7-4190-93fe-995ece656ee3\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " Oct 30 22:18:21.712496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.712382 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\") pod \"fc15dd84-b0d7-4190-93fe-995ece656ee3\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " Oct 30 22:18:21.712496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.712466 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\" (UniqueName: \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\") pod \"fc15dd84-b0d7-4190-93fe-995ece656ee3\" (UID: \"fc15dd84-b0d7-4190-93fe-995ece656ee3\") " Oct 30 22:18:21.712709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.712685 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca" pod "fc15dd84-b0d7-4190-93fe-995ece656ee3" (UID: "fc15dd84-b0d7-4190-93fe-995ece656ee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:21.713309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.713282 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:21.714815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.714729 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca" pod "fc15dd84-b0d7-4190-93fe-995ece656ee3" (UID: "fc15dd84-b0d7-4190-93fe-995ece656ee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:21.714815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.714792 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g" pod "fc15dd84-b0d7-4190-93fe-995ece656ee3" (UID: "fc15dd84-b0d7-4190-93fe-995ece656ee3"). InnerVolumeSpecName "kube-api-access-4t49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:21.814556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.814528 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc15dd84-b0d7-4190-93fe-995ece656ee3-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:21.814556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:21.814556 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t49g\" (UniqueName: \"kubernetes.io/projected/fc15dd84-b0d7-4190-93fe-995ece656ee3-kube-api-access-4t49g\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.419044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:22.418957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" Oct 30 22:18:22.419044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:22.418976 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp" event={"ID":"fc15dd84-b0d7-4190-93fe-995ece656ee3","Type":"ContainerDied","Data":"f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c"} Oct 30 22:18:22.419044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:22.419021 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ad57441094fbc1aa98b49e5611bb0ad2ed1bf230724d16e8b1a84bdfb69b4c" Oct 30 22:18:24.168531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.168491 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:18:24.171103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.169996 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fc15dd84-b0d7-4190-93fe-995ece656ee3" containerName="revoke-keys" Oct 30 22:18:24.171103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.170016 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc15dd84-b0d7-4190-93fe-995ece656ee3" containerName="revoke-keys" Oct 30 22:18:24.171103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.170122 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc15dd84-b0d7-4190-93fe-995ece656ee3" containerName="revoke-keys" Oct 30 22:18:24.174203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.174180 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.231664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.231633 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:18:24.244640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.242518 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xwq\" (UniqueName: \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.244640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.242999 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.291258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.275919 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea4ac3ec-dockercfg-bgvfl\"" Oct 30 22:18:24.312906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.277017 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea4ac3ec\"" Oct 30 22:18:24.358016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.347754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.358016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.347892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xwq\" (UniqueName: \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.361832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.361796 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.411924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.411789 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xwq\" (UniqueName: \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\") pod \"payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.493091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.493012 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:24.875346 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:24.875313 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb69f625_b439_4e6a_a4fd_da1f529b71f6.slice/crio-a6473c8bcb2d47f29648a86af360fa7f60b78e760f3bb6e9f0ee4a9c15e614ea WatchSource:0}: Error finding container a6473c8bcb2d47f29648a86af360fa7f60b78e760f3bb6e9f0ee4a9c15e614ea: Status 404 returned error can't find the container with id a6473c8bcb2d47f29648a86af360fa7f60b78e760f3bb6e9f0ee4a9c15e614ea Oct 30 22:18:24.877088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:24.877010 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:18:25.463085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:25.463049 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" event={"ID":"eb69f625-b439-4e6a-a4fd-da1f529b71f6","Type":"ContainerStarted","Data":"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee"} Oct 30 22:18:25.463465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:25.463104 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" event={"ID":"eb69f625-b439-4e6a-a4fd-da1f529b71f6","Type":"ContainerStarted","Data":"a6473c8bcb2d47f29648a86af360fa7f60b78e760f3bb6e9f0ee4a9c15e614ea"} Oct 30 22:18:25.463465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:25.463134 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:25.588832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:25.588775 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" podStartSLOduration=1.5887619069999999 podStartE2EDuration="1.588761907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:25.586032748 +0000 UTC m=+1433.554852313" watchObservedRunningTime="2026-10-30 22:18:25.588761907 +0000 UTC m=+1433.557581471" Oct 30 22:18:26.642606 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:18:26.642559 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:18:29.075476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.075437 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:18:29.081926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.081903 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.126073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.126047 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kp7\" (UniqueName: \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.126192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.126108 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.136709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.136680 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:18:29.181111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.181084 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4266be31\"" Oct 30 22:18:29.181298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.181097 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4266be31-dockercfg-5d9mk\"" Oct 30 22:18:29.268604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.232702 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.276600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.273064 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kp7\" (UniqueName: \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.280787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.267527 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.389741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.388791 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kp7\" (UniqueName: \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\") pod \"payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.402710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.402672 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:29.803941 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:18:29.803909 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02f70b0_2601_4dda_acd3_3aedc1cbee0a.slice/crio-978c4e7dd79e0ec02c1cd7ffc875ce9bd46b105493a8b114786f3edae70402d5 WatchSource:0}: Error finding container 978c4e7dd79e0ec02c1cd7ffc875ce9bd46b105493a8b114786f3edae70402d5: Status 404 returned error can't find the container with id 978c4e7dd79e0ec02c1cd7ffc875ce9bd46b105493a8b114786f3edae70402d5 Oct 30 22:18:29.805526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:29.805503 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:18:30.404226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:30.404192 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:18:30.513923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:30.513888 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" event={"ID":"b02f70b0-2601-4dda-acd3-3aedc1cbee0a","Type":"ContainerStarted","Data":"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770"} Oct 30 22:18:30.514074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:30.513943 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" event={"ID":"b02f70b0-2601-4dda-acd3-3aedc1cbee0a","Type":"ContainerStarted","Data":"978c4e7dd79e0ec02c1cd7ffc875ce9bd46b105493a8b114786f3edae70402d5"} Oct 30 22:18:30.514074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:30.513971 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:30.652491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:30.652442 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" podStartSLOduration=1.652427629 podStartE2EDuration="1.652427629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:30.648633047 +0000 UTC m=+1438.617452613" watchObservedRunningTime="2026-10-30 22:18:30.652427629 +0000 UTC m=+1438.621247193" Oct 30 22:18:35.436006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.435957 4292 scope.go:122] "RemoveContainer" containerID="848feba548c63fe53dd46ec9162c8d5b376843df27e70457d5858a6a3106eb1d" Oct 30 22:18:35.447144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.447121 4292 scope.go:122] "RemoveContainer" containerID="f4235624a0d0122937771026c16092f1847a791787b7525a99d12cebad9ca2dd" Oct 30 22:18:35.456881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.456861 4292 scope.go:122] "RemoveContainer" containerID="dc5ffeed3f2d616aa43a3f3749d7bdba7abbd6af307c8dc6fb24795ff1f1ffa6" Oct 30 22:18:35.466365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.466338 4292 scope.go:122] "RemoveContainer" containerID="487bcd3f92a4378c41faacbae4e44920422cfafed230fb837807f3d990e5dd9e" Oct 30 22:18:35.475524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.475504 4292 scope.go:122] "RemoveContainer" containerID="521be69df7d016f0e5a20ff08abbbf6b27f25210782b558b025ef458aff779d3" Oct 30 22:18:35.484491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.484474 4292 scope.go:122] "RemoveContainer" containerID="bf8a6523e849cad01c801010df0fafadcc00a57812fe7bb3d02e5c5ec112b6b3" Oct 30 22:18:35.493790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.493774 4292 scope.go:122] "RemoveContainer" containerID="c3d90df04ecda4a3220e1fa0f8c1d18225c2e303c985118ff521e9e23b43eaf3" Oct 30 22:18:35.502749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.502730 4292 scope.go:122] "RemoveContainer" containerID="29a3ee0f7dadfa7ce4ea7107b96a82a7e69155de85210338447cfcbd3b0df739" Oct 30 22:18:35.513192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.513161 4292 scope.go:122] "RemoveContainer" containerID="b8c97452d4a0c205d64520a583451497157136f7d5b45bf20f6f8c6a330dc60d" Oct 30 22:18:35.522467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.522450 4292 scope.go:122] "RemoveContainer" containerID="61e8c27e8fb82a945f372f44613c27a9efa0cace06cffc51df70ac677f3445bb" Oct 30 22:18:35.531303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.531284 4292 scope.go:122] "RemoveContainer" containerID="fc625de954b3bb49e7b1992f49e1f5abd75be8b10bfd488ac965d27118ac0738" Oct 30 22:18:35.540533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.540512 4292 scope.go:122] "RemoveContainer" containerID="823bbed3ddc812011f571243b7a6ac1ad3e11057c0db5abec394802f9f046ee0" Oct 30 22:18:35.552703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:35.552684 4292 scope.go:122] "RemoveContainer" containerID="fb2ecdaf420cdbb567e7f309cc6cefead413c11ccab0a00fb61babbcdb1ba287" Oct 30 22:18:36.475112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:36.475083 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:18:41.524956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:41.524923 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:18:51.595599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:51.580663 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:51.612601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:51.610037 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:51.612601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:51.610442 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 22:18:52.194600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.194127 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.229598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.228793 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.395717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.395669 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.444743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.444646 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.530566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.530519 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.590603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.587642 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:18:52.590603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.587881 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:18:52.602053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.602024 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.610674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.610639 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:52.611086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.611068 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:18:52.634105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.634061 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:18:52.791815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:18:52.791732 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:18:56.641012 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:18:56.640971 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:19:21.034063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.034023 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:19:21.040505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.040486 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.081963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.081938 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.081982 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082105 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082210 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082345 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082396 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082377 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082466 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqnn\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.082624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.082533 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.148330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.148293 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:19:21.183718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.183682 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.183962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.183939 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184431 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184483 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184508 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqnn\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184613 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184774 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184829 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.184980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.184937 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.185146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.185126 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.185337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.185315 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.185414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.185336 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.186194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.186126 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.188473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.188451 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.194332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.194304 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.247656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.247634 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.247847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.247829 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqnn\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\") pod \"e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.354405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.354383 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:21.791310 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:19:21.791279 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669dbe44_e54c_46ef_8c21_0da05d534351.slice/crio-027d59de63717d903a5a052b8993e7a449f3c30876dcbeb9a148bc5d616cd449 WatchSource:0}: Error finding container 027d59de63717d903a5a052b8993e7a449f3c30876dcbeb9a148bc5d616cd449: Status 404 returned error can't find the container with id 027d59de63717d903a5a052b8993e7a449f3c30876dcbeb9a148bc5d616cd449 Oct 30 22:19:21.792722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.792695 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:19:21.794302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.794271 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:21.794394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.794335 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:21.794394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:21.794364 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:22.167390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.167346 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" event={"ID":"669dbe44-e54c-46ef-8c21-0da05d534351","Type":"ContainerStarted","Data":"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3"} Oct 30 22:19:22.168142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.167417 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" event={"ID":"669dbe44-e54c-46ef-8c21-0da05d534351","Type":"ContainerStarted","Data":"027d59de63717d903a5a052b8993e7a449f3c30876dcbeb9a148bc5d616cd449"} Oct 30 22:19:22.354771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.354737 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:22.356718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.356686 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:22.356840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.356738 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:22.406432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.406362 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podStartSLOduration=2.406340722 podStartE2EDuration="2.406340722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:22.351424215 +0000 UTC m=+1490.320243801" watchObservedRunningTime="2026-10-30 22:19:22.406340722 +0000 UTC m=+1490.375160293" Oct 30 22:19:22.448006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.447919 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:19:22.453268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.453244 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.467399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.467379 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:19:22.471789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.471757 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.517605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.510144 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.517901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.517148 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9qrfj\"" Oct 30 22:19:22.518036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.517913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.517962 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518036 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518197 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518434 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.518558 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.518544 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4k9\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.524907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.524882 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:19:22.620817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.620707 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.621020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.620910 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.621780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621749 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.621912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621845 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxmf\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.621912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621876 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621927 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621962 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.621986 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622021 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622090 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622120 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622157 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622227 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622319 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622389 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622412 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622461 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.622612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4k9\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.622612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.622585 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.623072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.623046 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.623333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.623299 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.624044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.624017 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.624261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.624237 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.626636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.626568 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.629443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.629419 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.641178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.641149 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:19:22.709508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.709406 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.718243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.718212 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4k9\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\") pod \"maas-default-gateway-openshift-default-76597d6855-5xmvv\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.724201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.723877 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.723956 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724031 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724085 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724154 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724238 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724307 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724415 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxmf\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.724548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.725341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.724994 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.725666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.725644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.725889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.725862 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.726070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.726048 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.726213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.726125 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.728557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.728530 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.728785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.728768 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.768705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.768685 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:22.820410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.820371 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:22.820707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:22.820687 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxmf\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\") pod \"maas-default-gateway-openshift-default-76597d6855-slvnp\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:23.109878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.109839 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:23.256728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.255435 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:19:23.270545 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:19:23.270512 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d28cc4d_575c_41d1_9587_3ecd6acc1a5e.slice/crio-ccb625b37a548469e3fbe2230a586cc214f7cb1b7b6f5f07cb9cfd22f1e42d59 WatchSource:0}: Error finding container ccb625b37a548469e3fbe2230a586cc214f7cb1b7b6f5f07cb9cfd22f1e42d59: Status 404 returned error can't find the container with id ccb625b37a548469e3fbe2230a586cc214f7cb1b7b6f5f07cb9cfd22f1e42d59 Oct 30 22:19:23.274007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.273970 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:23.274210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.274192 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:23.274352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.274336 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:23.355062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.355033 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:23.355254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.355077 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:23.516431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.516398 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:19:23.516714 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:19:23.516678 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c62adc_bb80_4507_9d92_744a3fa751ab.slice/crio-b38a12143b9c57b3cc4e0223a472d866d9d10a5e3d43760ba6f362902201db05 WatchSource:0}: Error finding container b38a12143b9c57b3cc4e0223a472d866d9d10a5e3d43760ba6f362902201db05: Status 404 returned error can't find the container with id b38a12143b9c57b3cc4e0223a472d866d9d10a5e3d43760ba6f362902201db05 Oct 30 22:19:23.519893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.519857 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:23.519959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.519940 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:23.520015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:23.519988 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:24.246512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.246464 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" event={"ID":"d8c62adc-bb80-4507-9d92-744a3fa751ab","Type":"ContainerStarted","Data":"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627"} Oct 30 22:19:24.246757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.246536 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" event={"ID":"d8c62adc-bb80-4507-9d92-744a3fa751ab","Type":"ContainerStarted","Data":"b38a12143b9c57b3cc4e0223a472d866d9d10a5e3d43760ba6f362902201db05"} Oct 30 22:19:24.248501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.248478 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" event={"ID":"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e","Type":"ContainerStarted","Data":"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2"} Oct 30 22:19:24.248629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.248524 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" event={"ID":"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e","Type":"ContainerStarted","Data":"ccb625b37a548469e3fbe2230a586cc214f7cb1b7b6f5f07cb9cfd22f1e42d59"} Oct 30 22:19:24.358818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.355695 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:24.358818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.355757 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:24.478143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.478091 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podStartSLOduration=2.478076652 podStartE2EDuration="2.478076652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:24.472859541 +0000 UTC m=+1492.441679107" watchObservedRunningTime="2026-10-30 22:19:24.478076652 +0000 UTC m=+1492.446896215" Oct 30 22:19:24.657435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.657364 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podStartSLOduration=2.657339719 podStartE2EDuration="2.657339719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:24.601215007 +0000 UTC m=+1492.570034571" watchObservedRunningTime="2026-10-30 22:19:24.657339719 +0000 UTC m=+1492.626159283" Oct 30 22:19:24.772586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.772532 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:24.774711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.774681 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:24.774846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:24.774733 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:25.114601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.114372 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:25.114601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.114499 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:25.114601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.114544 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:25.362615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.361121 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:25.362615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.361184 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:25.770042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.770004 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:25.770257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:25.770072 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:26.112598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.111552 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:26.112598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.111636 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:26.358600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.356937 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:26.358600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.357000 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:26.758782 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:19:26.758738 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:19:26.769753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.769322 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:26.769753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:26.769380 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:27.117163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.117117 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:27.117465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.117423 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:27.355033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.354991 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:27.355319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.355285 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:27.769728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.769690 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:27.770231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:27.769844 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:28.117379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.117342 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:28.117701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.117668 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:28.355040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.354997 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:19:28.355239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.355047 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:19:28.769398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.769355 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:28.769632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:28.769410 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:29.123028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.122991 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:29.123652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.123607 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:29.745609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.745540 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:29.746419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.746395 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:29.747088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.747067 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:19:29.769590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.769549 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:29.769714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:29.769602 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:30.111367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:30.110914 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:30.111367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:30.110981 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:30.769857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:30.769817 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:19:30.770262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:30.769882 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:19:31.110733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:31.110698 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:19:31.110889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:31.110763 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:19:32.109317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.109281 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:32.111735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.111710 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:32.113419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.113396 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:19:32.376255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.376170 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:32.376637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.376611 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:32.377395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:32.377374 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:19:35.655248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:35.655213 4292 scope.go:122] "RemoveContainer" containerID="8a34e4e0b69dcfbb39b51b7dee49e9fd2e89c1e048bea7520a6f263ed34c42ae" Oct 30 22:19:35.666019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:35.665993 4292 scope.go:122] "RemoveContainer" containerID="686e7d18d33f54b8c692598700e755715cf9df9743e01682627959375e954510" Oct 30 22:19:41.713708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:41.713670 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:41.714217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:41.713727 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:44.526308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.526275 4292 generic.go:356] "Generic (PLEG): container finished" podID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerID="eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a" exitCode=0 Oct 30 22:19:44.526743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.526349 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerDied","Data":"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a"} Oct 30 22:19:44.526793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.526757 4292 scope.go:122] "RemoveContainer" containerID="eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a" Oct 30 22:19:44.528283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528261 4292 generic.go:356] "Generic (PLEG): container finished" podID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerID="0faa4ecfb13d8995fc1fca2164be136c2575747e499b3a725b3b39f03f3da0f1" exitCode=0 Oct 30 22:19:44.528385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528330 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerDied","Data":"0faa4ecfb13d8995fc1fca2164be136c2575747e499b3a725b3b39f03f3da0f1"} Oct 30 22:19:44.528385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528353 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:44.528464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528421 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:44.528464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528454 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:44.528676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.528661 4292 scope.go:122] "RemoveContainer" containerID="0faa4ecfb13d8995fc1fca2164be136c2575747e499b3a725b3b39f03f3da0f1" Oct 30 22:19:44.530100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.530072 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:44.530166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.530148 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:44.530210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:44.530193 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:19:45.576164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.576121 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerStarted","Data":"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4"} Oct 30 22:19:45.579233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.579196 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerStarted","Data":"ecf61e95d1d5c2359de6f575a6aea07b35ace0e2f943a9ed8a43cd47142955a0"} Oct 30 22:19:45.839745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.839203 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:19:45.839745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.839326 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:45.839745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.839428 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:45.849278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.848701 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:19:45.855301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.855181 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:45.855301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:45.855236 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:46.836012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:46.835972 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:46.836510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:46.836301 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:46.844239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:46.844214 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:46.844369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:46.844256 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:47.827601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:47.824328 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:47.827601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:47.824409 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:47.844296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:47.844264 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:47.845083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:47.845053 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:48.825607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:48.821954 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:48.825607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:48.822013 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:48.844797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:48.844760 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:48.845143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:48.844820 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:49.823600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:49.817757 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:19:49.823600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:49.820021 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:49.823600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:49.820070 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:49.844563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:49.844530 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:49.844563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:49.844595 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:50.731254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.731213 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b"] Oct 30 22:19:50.736996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.736971 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.744828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.744801 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.744954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.744836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.744954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.744901 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc5kg\" (UniqueName: \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.757475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.757450 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b"] Oct 30 22:19:50.808271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.808228 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:19:50.808494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.808313 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:50.808494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.808376 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:50.828598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.822943 4292 patch_prober.go:29] interesting pod/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:19:50.828598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.823029 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:19:50.841719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.841687 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:50.843999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.843971 4292 patch_prober.go:29] interesting pod/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:19:50.844144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.844014 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:19:50.846869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.846841 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.846997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.846878 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.846997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.846953 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc5kg\" (UniqueName: \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.847708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.847688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.849296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.849266 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:50.910204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:50.910161 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc5kg\" (UniqueName: \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\") pod \"maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:51.055330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.055296 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:51.436337 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:19:51.436307 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4724a4d8_8b88_45c8_9f46_d679ee382581.slice/crio-c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e WatchSource:0}: Error finding container c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e: Status 404 returned error can't find the container with id c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e Oct 30 22:19:51.439690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.439664 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b"] Oct 30 22:19:51.686910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.686827 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" event={"ID":"4724a4d8-8b88-45c8-9f46-d679ee382581","Type":"ContainerStarted","Data":"c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e"} Oct 30 22:19:51.824869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.824845 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:19:51.825388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.825365 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:19:51.826045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.826028 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:19:51.848745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.848718 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:19:51.849788 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:51.849769 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:19:52.697323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:52.697241 4292 generic.go:356] "Generic (PLEG): container finished" podID="4724a4d8-8b88-45c8-9f46-d679ee382581" containerID="1630fefefe26f2a65e9fdc28f969163ee7364bd8f18d16010fa491e0aa3ebebc" exitCode=0 Oct 30 22:19:52.697323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:52.697270 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" event={"ID":"4724a4d8-8b88-45c8-9f46-d679ee382581","Type":"ContainerDied","Data":"1630fefefe26f2a65e9fdc28f969163ee7364bd8f18d16010fa491e0aa3ebebc"} Oct 30 22:19:53.887768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.887738 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:53.994843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.994792 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\" (UniqueName: \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\") pod \"4724a4d8-8b88-45c8-9f46-d679ee382581\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " Oct 30 22:19:53.994945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.994872 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\") pod \"4724a4d8-8b88-45c8-9f46-d679ee382581\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " Oct 30 22:19:53.995045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.995033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\") pod \"4724a4d8-8b88-45c8-9f46-d679ee382581\" (UID: \"4724a4d8-8b88-45c8-9f46-d679ee382581\") " Oct 30 22:19:53.995247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.995214 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca" pod "4724a4d8-8b88-45c8-9f46-d679ee382581" (UID: "4724a4d8-8b88-45c8-9f46-d679ee382581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:53.995549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.995529 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:53.996993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.996966 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca" pod "4724a4d8-8b88-45c8-9f46-d679ee382581" (UID: "4724a4d8-8b88-45c8-9f46-d679ee382581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:53.997299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:53.997281 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg" pod "4724a4d8-8b88-45c8-9f46-d679ee382581" (UID: "4724a4d8-8b88-45c8-9f46-d679ee382581"). InnerVolumeSpecName "kube-api-access-gc5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:54.097292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:54.097260 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4724a4d8-8b88-45c8-9f46-d679ee382581-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:54.097292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:54.097290 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc5kg\" (UniqueName: \"kubernetes.io/projected/4724a4d8-8b88-45c8-9f46-d679ee382581-kube-api-access-gc5kg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:54.716519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:54.716487 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" Oct 30 22:19:54.716722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:54.716514 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b" event={"ID":"4724a4d8-8b88-45c8-9f46-d679ee382581","Type":"ContainerDied","Data":"c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e"} Oct 30 22:19:54.716722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:54.716568 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c848db3afa770607b02a6c76673b09f5cf93c9c412274720092c112bd1b1573e" Oct 30 22:19:56.649743 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:19:56.649706 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:19:59.578911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.578877 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx"] Oct 30 22:19:59.579641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.579622 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4724a4d8-8b88-45c8-9f46-d679ee382581" containerName="revoke-keys" Oct 30 22:19:59.579641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.579638 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4724a4d8-8b88-45c8-9f46-d679ee382581" containerName="revoke-keys" Oct 30 22:19:59.579798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.579743 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4724a4d8-8b88-45c8-9f46-d679ee382581" containerName="revoke-keys" Oct 30 22:19:59.583585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.583558 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.642909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.642887 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx"] Oct 30 22:19:59.671412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.671384 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.671561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.671465 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7cs\" (UniqueName: \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.671713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.671686 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.685993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.685970 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:59.686113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.686049 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:59.687892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.687862 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:59.688010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.687945 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:19:59.773714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.773686 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7cs\" (UniqueName: \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.773868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.773848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.774068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.774032 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.774698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.774680 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.776319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.776296 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.837126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.837071 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7cs\" (UniqueName: \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\") pod \"maas-api-revoke-sub-334aaa76-m5dkx\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:19:59.897377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:19:59.897356 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:20:00.282523 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:00.282494 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9696683c_bc35_48a7_b03f_763f52cbf872.slice/crio-d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566 WatchSource:0}: Error finding container d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566: Status 404 returned error can't find the container with id d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566 Oct 30 22:20:00.285142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:00.285116 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx"] Oct 30 22:20:00.770868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:00.770832 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" event={"ID":"9696683c-bc35-48a7-b03f-763f52cbf872","Type":"ContainerStarted","Data":"a56da57c041f93c88bf5af577fa5251d26680ab6e264c4b36b3ccb21e7547441"} Oct 30 22:20:00.771332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:00.770924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" event={"ID":"9696683c-bc35-48a7-b03f-763f52cbf872","Type":"ContainerStarted","Data":"d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566"} Oct 30 22:20:00.908648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:00.908133 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" podStartSLOduration=1.90811085 podStartE2EDuration="1.90811085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:00.896230902 +0000 UTC m=+1528.865050467" watchObservedRunningTime="2026-10-30 22:20:00.90811085 +0000 UTC m=+1528.876930415" Oct 30 22:20:01.780607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:01.780551 4292 generic.go:356] "Generic (PLEG): container finished" podID="9696683c-bc35-48a7-b03f-763f52cbf872" containerID="a56da57c041f93c88bf5af577fa5251d26680ab6e264c4b36b3ccb21e7547441" exitCode=0 Oct 30 22:20:01.781025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:01.780651 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" event={"ID":"9696683c-bc35-48a7-b03f-763f52cbf872","Type":"ContainerDied","Data":"a56da57c041f93c88bf5af577fa5251d26680ab6e264c4b36b3ccb21e7547441"} Oct 30 22:20:02.983833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:02.983803 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:20:03.131434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.131397 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\" (UniqueName: \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\") pod \"9696683c-bc35-48a7-b03f-763f52cbf872\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " Oct 30 22:20:03.131695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.131494 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\") pod \"9696683c-bc35-48a7-b03f-763f52cbf872\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " Oct 30 22:20:03.131695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.131657 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\") pod \"9696683c-bc35-48a7-b03f-763f52cbf872\" (UID: \"9696683c-bc35-48a7-b03f-763f52cbf872\") " Oct 30 22:20:03.132016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.131987 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca" pod "9696683c-bc35-48a7-b03f-763f52cbf872" (UID: "9696683c-bc35-48a7-b03f-763f52cbf872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:03.132741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.132716 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.133786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.133746 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs" pod "9696683c-bc35-48a7-b03f-763f52cbf872" (UID: "9696683c-bc35-48a7-b03f-763f52cbf872"). InnerVolumeSpecName "kube-api-access-jp7cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:03.136636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.136614 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca" pod "9696683c-bc35-48a7-b03f-763f52cbf872" (UID: "9696683c-bc35-48a7-b03f-763f52cbf872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:03.235114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.235076 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp7cs\" (UniqueName: \"kubernetes.io/projected/9696683c-bc35-48a7-b03f-763f52cbf872-kube-api-access-jp7cs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.235114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.235110 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9696683c-bc35-48a7-b03f-763f52cbf872-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.507377 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.507297 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:20:03.798781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.798756 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" Oct 30 22:20:03.799000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.798796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx" event={"ID":"9696683c-bc35-48a7-b03f-763f52cbf872","Type":"ContainerDied","Data":"d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566"} Oct 30 22:20:03.806035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:03.806005 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d044e6b81a70e8d02218535b523c901cfbcf4d8c169d67561b86c877dd84b566" Oct 30 22:20:04.032219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.032184 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs"] Oct 30 22:20:04.033186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.033163 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9696683c-bc35-48a7-b03f-763f52cbf872" containerName="revoke-keys" Oct 30 22:20:04.033240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.033190 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9696683c-bc35-48a7-b03f-763f52cbf872" containerName="revoke-keys" Oct 30 22:20:04.033496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.033475 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9696683c-bc35-48a7-b03f-763f52cbf872" containerName="revoke-keys" Oct 30 22:20:04.046054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.046020 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.097010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.096973 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs"] Oct 30 22:20:04.118138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.118100 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:04.118336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.118185 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:04.118336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.118311 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:20:04.138540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.138508 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:04.175940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.175904 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtrks\" (UniqueName: \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.176199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.176161 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.176306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.176243 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.277179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.277124 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtrks\" (UniqueName: \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.277179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.277185 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.277456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.277240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.277934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.277910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.279650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.279622 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.340845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.340789 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtrks\" (UniqueName: \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\") pod \"maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.380120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.380085 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:04.762312 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:04.762277 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74a9df6_65e6_4366_ba2e_b8189859f7a0.slice/crio-e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b WatchSource:0}: Error finding container e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b: Status 404 returned error can't find the container with id e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b Oct 30 22:20:04.764297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.764275 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs"] Oct 30 22:20:04.809756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:04.809730 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" event={"ID":"e74a9df6-65e6-4366-ba2e-b8189859f7a0","Type":"ContainerStarted","Data":"e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b"} Oct 30 22:20:05.826286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:05.826243 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" event={"ID":"e74a9df6-65e6-4366-ba2e-b8189859f7a0","Type":"ContainerStarted","Data":"72b7f8f541ee06a459b6df38585bd030f2e827cfe379d3d77efac9b153bf9b90"} Oct 30 22:20:05.948963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:05.948902 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" podStartSLOduration=2.948884062 podStartE2EDuration="2.948884062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:05.945536308 +0000 UTC m=+1533.914355885" watchObservedRunningTime="2026-10-30 22:20:05.948884062 +0000 UTC m=+1533.917703627" Oct 30 22:20:06.069666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:06.069631 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz"] Oct 30 22:20:06.075046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:06.075017 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12a4058-9wzwz"] Oct 30 22:20:06.149456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:06.149386 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f745c6f-c609-4c82-b742-7a6afe3c52df" path="/var/lib/kubelet/pods/8f745c6f-c609-4c82-b742-7a6afe3c52df/volumes" Oct 30 22:20:06.868373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:06.867049 4292 generic.go:356] "Generic (PLEG): container finished" podID="e74a9df6-65e6-4366-ba2e-b8189859f7a0" containerID="72b7f8f541ee06a459b6df38585bd030f2e827cfe379d3d77efac9b153bf9b90" exitCode=0 Oct 30 22:20:06.868373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:06.867273 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" event={"ID":"e74a9df6-65e6-4366-ba2e-b8189859f7a0","Type":"ContainerDied","Data":"72b7f8f541ee06a459b6df38585bd030f2e827cfe379d3d77efac9b153bf9b90"} Oct 30 22:20:08.070676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.070652 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:08.161897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.161835 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\") pod \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " Oct 30 22:20:08.162027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.161960 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\" (UniqueName: \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\") pod \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " Oct 30 22:20:08.162027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.161987 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\") pod \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\" (UID: \"e74a9df6-65e6-4366-ba2e-b8189859f7a0\") " Oct 30 22:20:08.162204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.162178 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca" pod "e74a9df6-65e6-4366-ba2e-b8189859f7a0" (UID: "e74a9df6-65e6-4366-ba2e-b8189859f7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:08.162658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.162639 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:08.164140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.164118 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks" pod "e74a9df6-65e6-4366-ba2e-b8189859f7a0" (UID: "e74a9df6-65e6-4366-ba2e-b8189859f7a0"). InnerVolumeSpecName "kube-api-access-dtrks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:08.164234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.164150 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca" pod "e74a9df6-65e6-4366-ba2e-b8189859f7a0" (UID: "e74a9df6-65e6-4366-ba2e-b8189859f7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:08.271159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.271128 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtrks\" (UniqueName: \"kubernetes.io/projected/e74a9df6-65e6-4366-ba2e-b8189859f7a0-kube-api-access-dtrks\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:08.271159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.271158 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74a9df6-65e6-4366-ba2e-b8189859f7a0-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:08.507968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.507882 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" podUID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" containerName="payload-processing" containerID="cri-o://b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d" gracePeriod=25 Oct 30 22:20:08.803316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.803288 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:20:08.880643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.880604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\") pod \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " Oct 30 22:20:08.880875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.880859 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\" (UniqueName: \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\") pod \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\" (UID: \"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d\") " Oct 30 22:20:08.880946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.880889 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume" pod "44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" (UID: "44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:08.881360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.881340 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:08.882755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.882733 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr" pod "44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" (UID: "44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d"). InnerVolumeSpecName "kube-api-access-4skrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:08.893556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.893536 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" Oct 30 22:20:08.893667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.893583 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs" event={"ID":"e74a9df6-65e6-4366-ba2e-b8189859f7a0","Type":"ContainerDied","Data":"e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b"} Oct 30 22:20:08.893729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.893666 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41899b8a7242cbf3876cbf481b637a0ddf3b497f92a225289bc2f2209e1ca1b" Oct 30 22:20:08.899699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.899673 4292 generic.go:356] "Generic (PLEG): container finished" podID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" containerID="b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d" exitCode=0 Oct 30 22:20:08.899798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.899722 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" Oct 30 22:20:08.899798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.899763 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" event={"ID":"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d","Type":"ContainerDied","Data":"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d"} Oct 30 22:20:08.899894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.899818 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z" event={"ID":"44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d","Type":"ContainerDied","Data":"aea3c5a3b1896803a432549d86942cc6b06ddbe2321bc4b0e6bc4c9c5933b255"} Oct 30 22:20:08.899894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.899845 4292 scope.go:122] "RemoveContainer" containerID="b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d" Oct 30 22:20:08.911161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.911140 4292 scope.go:122] "RemoveContainer" containerID="b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d" Oct 30 22:20:08.911429 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:08.911410 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d\": container with ID starting with b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d not found: ID does not exist" containerID="b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d" Oct 30 22:20:08.911477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.911436 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d"} err="failed to get container status \"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d\": rpc error: code = NotFound desc = could not find container \"b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d\": container with ID starting with b656307174c6e45e09de35abf871fa3cd84ad6ec91ba9b7da4ec19b931072f1d not found: ID does not exist" Oct 30 22:20:08.983553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:08.983517 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skrr\" (UniqueName: \"kubernetes.io/projected/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d-kube-api-access-4skrr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:09.042555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:09.042525 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:20:09.050099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:09.050077 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73fb6577-545dd5cb6f-szh9z"] Oct 30 22:20:10.148526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.148493 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" path="/var/lib/kubelet/pods/44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d/volumes" Oct 30 22:20:10.923451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.923415 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt"] Oct 30 22:20:10.924205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924187 4292 cpu_manager.go:414] "containerMap: removing container" podUID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" containerName="payload-processing" Oct 30 22:20:10.924252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924207 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" containerName="payload-processing" Oct 30 22:20:10.924252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924234 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e74a9df6-65e6-4366-ba2e-b8189859f7a0" containerName="revoke-keys" Oct 30 22:20:10.924252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924240 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74a9df6-65e6-4366-ba2e-b8189859f7a0" containerName="revoke-keys" Oct 30 22:20:10.924378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924369 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b1fd1d-c5cd-445a-ae35-11f0b50c3e8d" containerName="payload-processing" Oct 30 22:20:10.924410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.924381 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74a9df6-65e6-4366-ba2e-b8189859f7a0" containerName="revoke-keys" Oct 30 22:20:10.929802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.929780 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:10.987222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:10.987194 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt"] Oct 30 22:20:11.009174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.009138 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.009393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.009367 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvhn\" (UniqueName: \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.009481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.009442 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.031143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.031113 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:11.031263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.031147 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:11.031263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.031191 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:11.033874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.033856 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:20:11.111481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.111456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.111615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.111545 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvhn\" (UniqueName: \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.111669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.111613 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.112241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.112221 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.113914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.113894 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.176046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.175993 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvhn\" (UniqueName: \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\") pod \"maas-api-revoke-keys-e2e-ait-4266be31-5z2wt\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.245415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.245393 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:11.627396 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:11.627360 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6423b2_6f12_4ef5_babd_af021ca9b9b1.slice/crio-5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed WatchSource:0}: Error finding container 5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed: Status 404 returned error can't find the container with id 5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed Oct 30 22:20:11.629053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.629028 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt"] Oct 30 22:20:11.929119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.929084 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" event={"ID":"ae6423b2-6f12-4ef5-babd-af021ca9b9b1","Type":"ContainerStarted","Data":"518fe94b2d374f3230eae4102c721e749babf38281f3057b690529e4537f19de"} Oct 30 22:20:11.929254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:11.929146 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" event={"ID":"ae6423b2-6f12-4ef5-babd-af021ca9b9b1","Type":"ContainerStarted","Data":"5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed"} Oct 30 22:20:12.050895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:12.050846 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" podStartSLOduration=2.050832446 podStartE2EDuration="2.050832446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:12.048809316 +0000 UTC m=+1540.017628890" watchObservedRunningTime="2026-10-30 22:20:12.050832446 +0000 UTC m=+1540.019652058" Oct 30 22:20:12.938830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:12.938794 4292 generic.go:356] "Generic (PLEG): container finished" podID="ae6423b2-6f12-4ef5-babd-af021ca9b9b1" containerID="518fe94b2d374f3230eae4102c721e749babf38281f3057b690529e4537f19de" exitCode=0 Oct 30 22:20:12.939262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:12.938871 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" event={"ID":"ae6423b2-6f12-4ef5-babd-af021ca9b9b1","Type":"ContainerDied","Data":"518fe94b2d374f3230eae4102c721e749babf38281f3057b690529e4537f19de"} Oct 30 22:20:14.131996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.131969 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:14.261955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.261875 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\") pod \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " Oct 30 22:20:14.262119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.261990 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\") pod \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " Oct 30 22:20:14.262119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.262022 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\" (UniqueName: \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\") pod \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\" (UID: \"ae6423b2-6f12-4ef5-babd-af021ca9b9b1\") " Oct 30 22:20:14.262427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.262399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca" pod "ae6423b2-6f12-4ef5-babd-af021ca9b9b1" (UID: "ae6423b2-6f12-4ef5-babd-af021ca9b9b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:14.263298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.263276 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:14.264402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.264373 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca" pod "ae6423b2-6f12-4ef5-babd-af021ca9b9b1" (UID: "ae6423b2-6f12-4ef5-babd-af021ca9b9b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:14.264655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.264631 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn" pod "ae6423b2-6f12-4ef5-babd-af021ca9b9b1" (UID: "ae6423b2-6f12-4ef5-babd-af021ca9b9b1"). InnerVolumeSpecName "kube-api-access-5nvhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:14.365105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.365066 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:14.365105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.365105 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvhn\" (UniqueName: \"kubernetes.io/projected/ae6423b2-6f12-4ef5-babd-af021ca9b9b1-kube-api-access-5nvhn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:14.956699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.956658 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" event={"ID":"ae6423b2-6f12-4ef5-babd-af021ca9b9b1","Type":"ContainerDied","Data":"5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed"} Oct 30 22:20:14.956866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.956738 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f043ae782de8bcd51fb67dddeb82d667e4b442511132da05deb533153de3aed" Oct 30 22:20:14.956866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:14.956668 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt" Oct 30 22:20:15.525304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:15.525268 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:20:20.525877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.525828 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" podUID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" containerName="payload-processing" containerID="cri-o://c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee" gracePeriod=25 Oct 30 22:20:20.817233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.817205 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:20:20.873855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.873829 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\") pod \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " Oct 30 22:20:20.873985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.873899 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\" (UniqueName: \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\") pod \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\" (UID: \"eb69f625-b439-4e6a-a4fd-da1f529b71f6\") " Oct 30 22:20:20.874167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.874142 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume" pod "eb69f625-b439-4e6a-a4fd-da1f529b71f6" (UID: "eb69f625-b439-4e6a-a4fd-da1f529b71f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:20.874514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.874499 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb69f625-b439-4e6a-a4fd-da1f529b71f6-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:20.876024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.876003 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq" pod "eb69f625-b439-4e6a-a4fd-da1f529b71f6" (UID: "eb69f625-b439-4e6a-a4fd-da1f529b71f6"). InnerVolumeSpecName "kube-api-access-t8xwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:20.976115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:20.976082 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xwq\" (UniqueName: \"kubernetes.io/projected/eb69f625-b439-4e6a-a4fd-da1f529b71f6-kube-api-access-t8xwq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:21.015857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.015822 4292 generic.go:356] "Generic (PLEG): container finished" podID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" containerID="c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee" exitCode=0 Oct 30 22:20:21.015973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.015871 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" Oct 30 22:20:21.015973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.015903 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" event={"ID":"eb69f625-b439-4e6a-a4fd-da1f529b71f6","Type":"ContainerDied","Data":"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee"} Oct 30 22:20:21.015973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.015958 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq" event={"ID":"eb69f625-b439-4e6a-a4fd-da1f529b71f6","Type":"ContainerDied","Data":"a6473c8bcb2d47f29648a86af360fa7f60b78e760f3bb6e9f0ee4a9c15e614ea"} Oct 30 22:20:21.016097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.015985 4292 scope.go:122] "RemoveContainer" containerID="c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee" Oct 30 22:20:21.028088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.028071 4292 scope.go:122] "RemoveContainer" containerID="c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee" Oct 30 22:20:21.028321 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:21.028302 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee\": container with ID starting with c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee not found: ID does not exist" containerID="c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee" Oct 30 22:20:21.028390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.028329 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee"} err="failed to get container status \"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee\": rpc error: code = NotFound desc = could not find container \"c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee\": container with ID starting with c342d02a95da0b12d988585926e7db91127157e263882a822bbe3237836b3aee not found: ID does not exist" Oct 30 22:20:21.193309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.193276 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:20:21.198561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:21.198532 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea4ac3ec-5b69fbdc7c-hmljq"] Oct 30 22:20:22.147459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:22.147427 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" path="/var/lib/kubelet/pods/eb69f625-b439-4e6a-a4fd-da1f529b71f6/volumes" Oct 30 22:20:22.558104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:22.558069 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:20:26.641652 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:26.641612 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:20:27.558539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.558497 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" podUID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" containerName="payload-pre-processing" containerID="cri-o://9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770" gracePeriod=25 Oct 30 22:20:27.893087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.893055 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:20:27.985456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.985413 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\" (UniqueName: \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\") pod \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " Oct 30 22:20:27.985871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.985847 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\") pod \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\" (UID: \"b02f70b0-2601-4dda-acd3-3aedc1cbee0a\") " Oct 30 22:20:27.986287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.986263 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume" pod "b02f70b0-2601-4dda-acd3-3aedc1cbee0a" (UID: "b02f70b0-2601-4dda-acd3-3aedc1cbee0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:27.986500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.986481 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:27.992024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:27.991990 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7" pod "b02f70b0-2601-4dda-acd3-3aedc1cbee0a" (UID: "b02f70b0-2601-4dda-acd3-3aedc1cbee0a"). InnerVolumeSpecName "kube-api-access-49kp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:28.088278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.088243 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kp7\" (UniqueName: \"kubernetes.io/projected/b02f70b0-2601-4dda-acd3-3aedc1cbee0a-kube-api-access-49kp7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.125778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.124137 4292 generic.go:356] "Generic (PLEG): container finished" podID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" containerID="9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770" exitCode=0 Oct 30 22:20:28.125778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.124359 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" event={"ID":"b02f70b0-2601-4dda-acd3-3aedc1cbee0a","Type":"ContainerDied","Data":"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770"} Oct 30 22:20:28.125778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.124413 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" event={"ID":"b02f70b0-2601-4dda-acd3-3aedc1cbee0a","Type":"ContainerDied","Data":"978c4e7dd79e0ec02c1cd7ffc875ce9bd46b105493a8b114786f3edae70402d5"} Oct 30 22:20:28.125778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.124445 4292 scope.go:122] "RemoveContainer" containerID="9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770" Oct 30 22:20:28.125778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.124632 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf" Oct 30 22:20:28.144162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.144141 4292 scope.go:122] "RemoveContainer" containerID="9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770" Oct 30 22:20:28.144712 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:28.144685 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770\": container with ID starting with 9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770 not found: ID does not exist" containerID="9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770" Oct 30 22:20:28.144859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.144836 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770"} err="failed to get container status \"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770\": rpc error: code = NotFound desc = could not find container \"9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770\": container with ID starting with 9523b95ed3a22d684e409b5f0235a29b68d99d5dc2dadb511b9823bd12900770 not found: ID does not exist" Oct 30 22:20:28.310860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.310824 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:20:28.321891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:28.320512 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4266be31-7f5fd6d76c-lsxqf"] Oct 30 22:20:30.147808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:30.147773 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" path="/var/lib/kubelet/pods/b02f70b0-2601-4dda-acd3-3aedc1cbee0a/volumes" Oct 30 22:20:37.064923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:37.064889 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r"] Oct 30 22:20:37.116345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:37.116312 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceaebf0-qhh5r"] Oct 30 22:20:38.148533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:38.148490 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a34412d-8052-47f4-8717-022337f08141" path="/var/lib/kubelet/pods/3a34412d-8052-47f4-8717-022337f08141/volumes" Oct 30 22:20:40.907530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:40.907495 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b"] Oct 30 22:20:40.951281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:40.951247 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-73fb6577-b2t2b"] Oct 30 22:20:40.956740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:40.956715 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:20:40.956941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:40.956919 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" containerID="cri-o://32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4" gracePeriod=30 Oct 30 22:20:42.150668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.150634 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4724a4d8-8b88-45c8-9f46-d679ee382581" path="/var/lib/kubelet/pods/4724a4d8-8b88-45c8-9f46-d679ee382581/volumes" Oct 30 22:20:42.863198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863165 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:20:42.863849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863834 4292 cpu_manager.go:414] "containerMap: removing container" podUID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" containerName="payload-processing" Oct 30 22:20:42.863905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863850 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" containerName="payload-processing" Oct 30 22:20:42.863905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863882 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ae6423b2-6f12-4ef5-babd-af021ca9b9b1" containerName="revoke-keys" Oct 30 22:20:42.863905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863887 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6423b2-6f12-4ef5-babd-af021ca9b9b1" containerName="revoke-keys" Oct 30 22:20:42.863905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863895 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" containerName="payload-pre-processing" Oct 30 22:20:42.863905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.863900 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" containerName="payload-pre-processing" Oct 30 22:20:42.864067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.864035 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6423b2-6f12-4ef5-babd-af021ca9b9b1" containerName="revoke-keys" Oct 30 22:20:42.864067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.864045 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02f70b0-2601-4dda-acd3-3aedc1cbee0a" containerName="payload-pre-processing" Oct 30 22:20:42.864067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.864056 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb69f625-b439-4e6a-a4fd-da1f529b71f6" containerName="payload-processing" Oct 30 22:20:42.869104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.869085 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:42.951009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.950973 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-05eb684c-gw-openshift-default-dockercfg-4dplf\"" Oct 30 22:20:42.999672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:42.983680 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.016732 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxrx\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.018741 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.018813 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.018915 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.018976 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.019047 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.019063 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.023597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.019843 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.061644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.053245 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:20:43.122340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122242 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122340 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122375 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxrx\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122552 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122715 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122776 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122784 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122899 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.122935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.122929 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.123217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.123191 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.123308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.123231 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.123402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.123384 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.123896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.123875 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.126044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.126023 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.127837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.127816 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.189409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.189376 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.189779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.189381 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxrx\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\") pod \"e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.484026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.483904 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:43.935024 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:43.934995 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7c79ed_c3a6_45d1_ae79_579ad961ecb4.slice/crio-0969503f0b6d163c095e2e6494fda6ba0ad7bd8d0e29d45eb10832c1d115f59d WatchSource:0}: Error finding container 0969503f0b6d163c095e2e6494fda6ba0ad7bd8d0e29d45eb10832c1d115f59d: Status 404 returned error can't find the container with id 0969503f0b6d163c095e2e6494fda6ba0ad7bd8d0e29d45eb10832c1d115f59d Oct 30 22:20:43.936301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.936275 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:20:43.937703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.937673 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:43.937780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.937739 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:43.937780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:43.937772 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:44.286898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:44.286811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerStarted","Data":"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777"} Oct 30 22:20:44.286898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:44.286872 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerStarted","Data":"0969503f0b6d163c095e2e6494fda6ba0ad7bd8d0e29d45eb10832c1d115f59d"} Oct 30 22:20:44.416667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:44.416618 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podStartSLOduration=2.416604119 podStartE2EDuration="2.416604119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:44.41261822 +0000 UTC m=+1572.381437784" watchObservedRunningTime="2026-10-30 22:20:44.416604119 +0000 UTC m=+1572.385423682" Oct 30 22:20:44.484643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:44.484616 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:45.490485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:45.490414 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:46.282137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.282104 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:20:46.311793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.311763 4292 generic.go:356] "Generic (PLEG): container finished" podID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerID="32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4" exitCode=0 Oct 30 22:20:46.311924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.311816 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" Oct 30 22:20:46.311924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.311837 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerDied","Data":"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4"} Oct 30 22:20:46.311924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.311890 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p" event={"ID":"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90","Type":"ContainerDied","Data":"b5303008dfb2356648c6ccf907c3615108c041518c6602e7effc231c141052f8"} Oct 30 22:20:46.311924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.311913 4292 scope.go:122] "RemoveContainer" containerID="32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4" Oct 30 22:20:46.312372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.312349 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:46.313288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.313272 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:20:46.322886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.322869 4292 scope.go:122] "RemoveContainer" containerID="eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a" Oct 30 22:20:46.332502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.332477 4292 scope.go:122] "RemoveContainer" containerID="32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4" Oct 30 22:20:46.332830 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:46.332808 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4\": container with ID starting with 32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4 not found: ID does not exist" containerID="32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4" Oct 30 22:20:46.332888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.332837 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4"} err="failed to get container status \"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4\": rpc error: code = NotFound desc = could not find container \"32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4\": container with ID starting with 32c9fb74c0858319b142ad22bd311b7a6016e2434c471efc73b44021ef5d24e4 not found: ID does not exist" Oct 30 22:20:46.332888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.332854 4292 scope.go:122] "RemoveContainer" containerID="eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a" Oct 30 22:20:46.333096 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:46.333077 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a\": container with ID starting with eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a not found: ID does not exist" containerID="eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a" Oct 30 22:20:46.333141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.333103 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a"} err="failed to get container status \"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a\": rpc error: code = NotFound desc = could not find container \"eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a\": container with ID starting with eaac1cb71210c22478bf56b4a9931ca25e1511b7a637554f6ab7ea3afbf1040a not found: ID does not exist" Oct 30 22:20:46.384499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384477 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384530 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384557 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384666 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384724 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384775 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384801 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.384936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384921 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.385037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384968 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:46.385037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.384953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.385037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385006 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\") pod \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\" (UID: \"be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90\") " Oct 30 22:20:46.385187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385029 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.385263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385240 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.385686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385667 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.385779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385691 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.385779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385704 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.385779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385716 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.385779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.385729 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.387231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.387202 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.387338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.387271 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "kube-api-access-ch5tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:46.387338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.387296 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:46.387484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.387462 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token" pod "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" (UID: "be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:46.487217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.487115 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.487217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.487143 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.487217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.487158 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.487217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.487173 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5tr\" (UniqueName: \"kubernetes.io/projected/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90-kube-api-access-ch5tr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.766622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.766585 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:20:46.771822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:46.771792 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-73fb6577-openshift-default-647f7988d7-l8q6p"] Oct 30 22:20:48.148718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:48.148684 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" path="/var/lib/kubelet/pods/be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90/volumes" Oct 30 22:20:49.999005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:49.998970 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:20:49.999924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:49.999902 4292 cpu_manager.go:414] "containerMap: removing container" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.000043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:49.999925 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.000160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.000145 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.000219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.000172 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.005989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.005964 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.070913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.070877 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:20:50.109164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.109129 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-05eb684c\"" Oct 30 22:20:50.109404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.109229 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-05eb684c-dockercfg-6q7nb\"" Oct 30 22:20:50.298595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.296598 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2tf\" (UniqueName: \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.298595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.296785 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.340812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.339092 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:20:50.340812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.340357 4292 cpu_manager.go:414] "containerMap: removing container" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.340812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.340381 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be692e1e-f0c3-4791-8f5b-e8e8cbd4ec90" containerName="istio-proxy" Oct 30 22:20:50.346175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.346151 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.347075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.347045 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:20:50.397896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.397852 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2tf\" (UniqueName: \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.398058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.398011 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.399213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.399184 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.461803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.461771 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2tf\" (UniqueName: \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\") pod \"payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.500231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.500188 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4t5\" (UniqueName: \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.500384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.500319 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.602463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.602388 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4t5\" (UniqueName: \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.602627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.602477 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.603146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.603124 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.622115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.622093 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:50.666157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.666129 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4t5\" (UniqueName: \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\") pod \"payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:50.666961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:50.666930 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:51.006761 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:51.006733 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf923105_2634_4a7c_93f4_a2efef0dd57f.slice/crio-259fa56a443c2c978c8b00962aaa340ff0441f7eca950dac58d575cd1bc356a2 WatchSource:0}: Error finding container 259fa56a443c2c978c8b00962aaa340ff0441f7eca950dac58d575cd1bc356a2: Status 404 returned error can't find the container with id 259fa56a443c2c978c8b00962aaa340ff0441f7eca950dac58d575cd1bc356a2 Oct 30 22:20:51.009660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.009620 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:20:51.117216 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:51.117187 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3991f2e3_1c48_416f_a37e_d814d9c64381.slice/crio-e988c31aa64bb5764dd9c97848d91b399bc2d0e00c437d01421235affe159363 WatchSource:0}: Error finding container e988c31aa64bb5764dd9c97848d91b399bc2d0e00c437d01421235affe159363: Status 404 returned error can't find the container with id e988c31aa64bb5764dd9c97848d91b399bc2d0e00c437d01421235affe159363 Oct 30 22:20:51.120228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.120202 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:20:51.370623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.370540 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" event={"ID":"bf923105-2634-4a7c-93f4-a2efef0dd57f","Type":"ContainerStarted","Data":"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de"} Oct 30 22:20:51.370854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.370634 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" event={"ID":"bf923105-2634-4a7c-93f4-a2efef0dd57f","Type":"ContainerStarted","Data":"259fa56a443c2c978c8b00962aaa340ff0441f7eca950dac58d575cd1bc356a2"} Oct 30 22:20:51.370986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.370966 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:20:51.382413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.382382 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" event={"ID":"3991f2e3-1c48-416f-a37e-d814d9c64381","Type":"ContainerStarted","Data":"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf"} Oct 30 22:20:51.382566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.382434 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" event={"ID":"3991f2e3-1c48-416f-a37e-d814d9c64381","Type":"ContainerStarted","Data":"e988c31aa64bb5764dd9c97848d91b399bc2d0e00c437d01421235affe159363"} Oct 30 22:20:51.382566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.382537 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:20:51.493491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.493436 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" podStartSLOduration=2.493421187 podStartE2EDuration="2.493421187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:51.491948499 +0000 UTC m=+1579.460768063" watchObservedRunningTime="2026-10-30 22:20:51.493421187 +0000 UTC m=+1579.462240751" Oct 30 22:20:51.634030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:51.633906 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" podStartSLOduration=1.633890954 podStartE2EDuration="1.633890954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:51.628051051 +0000 UTC m=+1579.596870618" watchObservedRunningTime="2026-10-30 22:20:51.633890954 +0000 UTC m=+1579.602710518" Oct 30 22:20:53.063503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.063460 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8"] Oct 30 22:20:53.115383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.115344 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9208411-qlvp8"] Oct 30 22:20:53.195812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.194715 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:20:53.201018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.200988 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.289042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.289006 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:20:53.370767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.370684 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.370767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.370739 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.370979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.370849 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.370979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.370908 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.371101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.371078 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwpv\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.371181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.371165 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.371241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.371204 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.371291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.371243 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.371437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.371423 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472522 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472607 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472657 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472722 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472784 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.472884 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwpv\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473067 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473106 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473473 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473705 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.473966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.473956 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.474281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.474235 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.474426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.474373 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.475862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.475826 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.477348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.477319 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.537636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.537557 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.537826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.537642 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwpv\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\") pod \"maas-default-gateway-openshift-default-76597d6855-jdf2g\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:53.741673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.735487 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs"] Oct 30 22:20:53.784604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.774782 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea4ac3ec-fq6vs"] Oct 30 22:20:53.819288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:53.819253 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:54.148328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.148293 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc9dd1b-d5ab-490e-8456-5021f9be4d1b" path="/var/lib/kubelet/pods/dcc9dd1b-d5ab-490e-8456-5021f9be4d1b/volumes" Oct 30 22:20:54.148814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.148798 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74a9df6-65e6-4366-ba2e-b8189859f7a0" path="/var/lib/kubelet/pods/e74a9df6-65e6-4366-ba2e-b8189859f7a0/volumes" Oct 30 22:20:54.262940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.262902 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:20:54.263223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.263187 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" containerID="cri-o://ecf61e95d1d5c2359de6f575a6aea07b35ace0e2f943a9ed8a43cd47142955a0" gracePeriod=30 Oct 30 22:20:54.283058 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:54.283033 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b538b4d_25a3_4ce6_905d_57411194abcd.slice/crio-14fb8f7c9cfa3e6e01f865b981dcf736e8cdad7041d52f73ac1c4a5dc823ea9d WatchSource:0}: Error finding container 14fb8f7c9cfa3e6e01f865b981dcf736e8cdad7041d52f73ac1c4a5dc823ea9d: Status 404 returned error can't find the container with id 14fb8f7c9cfa3e6e01f865b981dcf736e8cdad7041d52f73ac1c4a5dc823ea9d Oct 30 22:20:54.285944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.285914 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:54.286028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.285979 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:54.286028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.286009 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:54.289426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.289398 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:20:54.416834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.416707 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" event={"ID":"4b538b4d-25a3-4ce6-905d-57411194abcd","Type":"ContainerStarted","Data":"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4"} Oct 30 22:20:54.416834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.416777 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" event={"ID":"4b538b4d-25a3-4ce6-905d-57411194abcd","Type":"ContainerStarted","Data":"14fb8f7c9cfa3e6e01f865b981dcf736e8cdad7041d52f73ac1c4a5dc823ea9d"} Oct 30 22:20:54.563597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.562806 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podStartSLOduration=1.562786815 podStartE2EDuration="1.562786815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:54.545465335 +0000 UTC m=+1582.514284900" watchObservedRunningTime="2026-10-30 22:20:54.562786815 +0000 UTC m=+1582.531606380" Oct 30 22:20:54.819960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:54.819928 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:20:55.131076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.130993 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt"] Oct 30 22:20:55.183647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.183613 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4266be31-5z2wt"] Oct 30 22:20:55.530515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.530437 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:20:55.535655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.535633 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.657710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657665 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.657895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657750 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxmx\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.657895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657780 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657925 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657963 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.657994 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.658166 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.658263 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.658375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.658335 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.660195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.660123 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-678b1628-openshift-default-dockercfg-jv2fq\"" Oct 30 22:20:55.661621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.661597 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.759849 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.759909 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.759940 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760087 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760166 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760221 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760260 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760305 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxmx\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.760595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.760328 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.796656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.761424 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.796656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.761703 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.796656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.761908 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.796656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.762172 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.796656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.762368 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.877953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.867622 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:20:55.877953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.867701 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:20:55.898197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.895252 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.900634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.900540 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxmx\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.914753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.913963 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:55.914753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:55.914549 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\") pod \"e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:56.150780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.150692 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6423b2-6f12-4ef5-babd-af021ca9b9b1" path="/var/lib/kubelet/pods/ae6423b2-6f12-4ef5-babd-af021ca9b9b1/volumes" Oct 30 22:20:56.152854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.152826 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:56.627446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.627414 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:20:56.648787 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:20:56.648752 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:20:56.655941 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:20:56.655917 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1df68f5_165d_4876_9379_27b2e703e508.slice/crio-26a02069353683fc5b811bf11b3fc3a5d694de767f2be0769f457d3803e04052 WatchSource:0}: Error finding container 26a02069353683fc5b811bf11b3fc3a5d694de767f2be0769f457d3803e04052: Status 404 returned error can't find the container with id 26a02069353683fc5b811bf11b3fc3a5d694de767f2be0769f457d3803e04052 Oct 30 22:20:56.658632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.658600 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:56.658714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.658666 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:56.658714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.658698 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:56.822962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.822920 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:20:56.823155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:56.823015 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:20:57.470416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:57.470381 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerStarted","Data":"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6"} Oct 30 22:20:57.470610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:57.470441 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerStarted","Data":"26a02069353683fc5b811bf11b3fc3a5d694de767f2be0769f457d3803e04052"} Oct 30 22:20:57.614641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:57.608002 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podStartSLOduration=2.60798284 podStartE2EDuration="2.60798284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:57.596105303 +0000 UTC m=+1585.564924868" watchObservedRunningTime="2026-10-30 22:20:57.60798284 +0000 UTC m=+1585.576802404" Oct 30 22:20:57.823350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:57.821889 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:20:57.823350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:57.821954 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:20:58.153932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.153853 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:58.159265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.159242 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:58.478834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.478757 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:58.479521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.479500 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:20:58.820120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.820087 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:20:58.820293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:58.820151 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:20:59.518968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.518926 4292 generic.go:356] "Generic (PLEG): container finished" podID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerID="ecf61e95d1d5c2359de6f575a6aea07b35ace0e2f943a9ed8a43cd47142955a0" exitCode=0 Oct 30 22:20:59.519457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.519037 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerDied","Data":"ecf61e95d1d5c2359de6f575a6aea07b35ace0e2f943a9ed8a43cd47142955a0"} Oct 30 22:20:59.519457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.519138 4292 scope.go:122] "RemoveContainer" containerID="0faa4ecfb13d8995fc1fca2164be136c2575747e499b3a725b3b39f03f3da0f1" Oct 30 22:20:59.640090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.640062 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:20:59.732904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.732870 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.732911 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.732934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.732967 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.732997 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733078 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733156 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:59.733240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733188 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733225 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\") pod \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\" (UID: \"3cb2bfca-4232-4aff-a804-a9cbd4d48e09\") " Oct 30 22:20:59.733373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733357 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:59.733908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733833 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:59.733908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733847 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.733908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733868 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.733908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733882 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:59.734179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.733994 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:59.736089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.736064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:59.736195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.736097 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "kube-api-access-rb6xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:59.736195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.736173 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:59.736307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.736272 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token" pod "3cb2bfca-4232-4aff-a804-a9cbd4d48e09" (UID: "3cb2bfca-4232-4aff-a804-a9cbd4d48e09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:59.820216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.820183 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:20:59.820370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.820238 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:20:59.835296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835266 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835298 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835315 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6xk\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-kube-api-access-rb6xk\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835329 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835344 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835358 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:59.835423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:20:59.835373 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb2bfca-4232-4aff-a804-a9cbd4d48e09-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:00.529345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.529260 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" Oct 30 22:21:00.529345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.529303 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q" event={"ID":"3cb2bfca-4232-4aff-a804-a9cbd4d48e09","Type":"ContainerDied","Data":"372b18ca32ec96142077857d1b9f2eaace2ee6a439d63320fc71a34e75392ba7"} Oct 30 22:21:00.529849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.529370 4292 scope.go:122] "RemoveContainer" containerID="ecf61e95d1d5c2359de6f575a6aea07b35ace0e2f943a9ed8a43cd47142955a0" Oct 30 22:21:00.764454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.764422 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:21:00.770627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.770559 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ea4ac3ec-openshift-default-547d64cdf5-lf99q"] Oct 30 22:21:00.823882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.823850 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:21:00.824120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:00.824092 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:21:01.819883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:01.819850 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:21:01.820248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:01.819899 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:21:02.150759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:02.150679 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" path="/var/lib/kubelet/pods/3cb2bfca-4232-4aff-a804-a9cbd4d48e09/volumes" Oct 30 22:21:02.399908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:02.399874 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:21:02.400093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:02.399971 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:21:03.635769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.635730 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:21:03.638301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.638268 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:21:03.639153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.639131 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:21:03.778003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.777972 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:21:03.778738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.778722 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.778795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.778739 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.778875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.778866 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.778908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.778881 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.784341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.784305 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:03.880492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.870697 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-678b1628-dockercfg-8pkv4\"" Oct 30 22:21:03.880492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.876803 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-678b1628\"" Oct 30 22:21:03.889616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.889498 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:21:03.931429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.931381 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvnt\" (UniqueName: \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:03.931724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.931449 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:03.963199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.963166 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:21:03.964495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.964460 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.964495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.964491 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb2bfca-4232-4aff-a804-a9cbd4d48e09" containerName="istio-proxy" Oct 30 22:21:03.972760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:03.972706 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.026632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.026597 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:21:04.032746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.032716 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.032901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.032754 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxrr\" (UniqueName: \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.032974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.032941 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvnt\" (UniqueName: \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:04.033027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.032989 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:04.034085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.034059 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:04.099657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.099620 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvnt\" (UniqueName: \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\") pod \"payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:04.106499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.106470 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:04.136091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.136052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.136235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.136102 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxrr\" (UniqueName: \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.137682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.137652 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.224680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.217367 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxrr\" (UniqueName: \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\") pod \"payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.302730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.302690 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:04.554482 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:21:04.550073 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a80a521_b11f_4fe0_a73c_c7aaffed564d.slice/crio-c31019d6193e8eefee3e23463b252e32732df9efa283c62ad3cff12a19b23ee5 WatchSource:0}: Error finding container c31019d6193e8eefee3e23463b252e32732df9efa283c62ad3cff12a19b23ee5: Status 404 returned error can't find the container with id c31019d6193e8eefee3e23463b252e32732df9efa283c62ad3cff12a19b23ee5 Oct 30 22:21:04.564802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.564775 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:21:04.583264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.583238 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" event={"ID":"5a80a521-b11f-4fe0-a73c-c7aaffed564d","Type":"ContainerStarted","Data":"c31019d6193e8eefee3e23463b252e32732df9efa283c62ad3cff12a19b23ee5"} Oct 30 22:21:04.757095 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:21:04.757065 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67caa40_2a33_44a8_80e0_ae9f8ecb2cc2.slice/crio-ff0b53bfbe03d7bffee188ef7f24e44c719b31626718c5dd85f794e3d5897bf4 WatchSource:0}: Error finding container ff0b53bfbe03d7bffee188ef7f24e44c719b31626718c5dd85f794e3d5897bf4: Status 404 returned error can't find the container with id ff0b53bfbe03d7bffee188ef7f24e44c719b31626718c5dd85f794e3d5897bf4 Oct 30 22:21:04.759076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:04.759053 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:21:05.593302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.593266 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" event={"ID":"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2","Type":"ContainerStarted","Data":"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b"} Oct 30 22:21:05.593462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.593321 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" event={"ID":"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2","Type":"ContainerStarted","Data":"ff0b53bfbe03d7bffee188ef7f24e44c719b31626718c5dd85f794e3d5897bf4"} Oct 30 22:21:05.593523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.593504 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:05.595116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.595092 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" event={"ID":"5a80a521-b11f-4fe0-a73c-c7aaffed564d","Type":"ContainerStarted","Data":"66e2687f7372d820747fc4dd4eaeed1ee00390de823130dfe49246d2558cc3d5"} Oct 30 22:21:05.595313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.595297 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:05.712705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.712661 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" podStartSLOduration=2.712648093 podStartE2EDuration="2.712648093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:05.71235476 +0000 UTC m=+1593.681174323" watchObservedRunningTime="2026-10-30 22:21:05.712648093 +0000 UTC m=+1593.681467657" Oct 30 22:21:05.861649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:05.860794 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" podStartSLOduration=2.860772987 podStartE2EDuration="2.860772987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:05.845277727 +0000 UTC m=+1593.814097291" watchObservedRunningTime="2026-10-30 22:21:05.860772987 +0000 UTC m=+1593.829592552" Oct 30 22:21:13.400852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.400818 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:21:13.405814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.405794 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.460453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.460422 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:21:13.460570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.460481 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:21:13.460659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.460625 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:21:13.487569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.487546 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.487697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.487629 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.487779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.487757 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.487925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.487905 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.487965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.487938 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.488119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.488089 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsd5n\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.488225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.488133 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.515080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.510082 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x7p2g\"" Oct 30 22:21:13.520881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.520856 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:21:13.589984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.589955 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590051 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590150 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590173 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590270 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsd5n\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590302 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590333 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.590859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.590830 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.592439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.592410 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.592549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.592515 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.592655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.592620 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.592715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.592654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.653318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.653261 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.654638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.654606 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsd5n\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lb4wk\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:13.720397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:13.720365 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:14.169095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:14.169063 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:21:14.187987 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:21:14.187961 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1687ba16_2f7f_42d3_96c9_2faa5e1be479.slice/crio-8640973ba7975ff4607102c2242577d2a7f76f05caa167571674b95e888a7358 WatchSource:0}: Error finding container 8640973ba7975ff4607102c2242577d2a7f76f05caa167571674b95e888a7358: Status 404 returned error can't find the container with id 8640973ba7975ff4607102c2242577d2a7f76f05caa167571674b95e888a7358 Oct 30 22:21:14.189982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:14.189961 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:21:14.690335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:14.690302 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" event={"ID":"1687ba16-2f7f-42d3-96c9-2faa5e1be479","Type":"ContainerStarted","Data":"8640973ba7975ff4607102c2242577d2a7f76f05caa167571674b95e888a7358"} Oct 30 22:21:16.607205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:16.607173 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:21:16.607704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:16.607250 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:21:17.063058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:17.063023 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94"] Oct 30 22:21:17.072519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:17.072491 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85fe4-d5s94"] Oct 30 22:21:18.037402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.037369 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:18.037712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.037437 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:18.149594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.149540 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f571ad-c298-4ce8-a013-92964a4b71d6" path="/var/lib/kubelet/pods/b3f571ad-c298-4ce8-a013-92964a4b71d6/volumes" Oct 30 22:21:18.737449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.737413 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" event={"ID":"1687ba16-2f7f-42d3-96c9-2faa5e1be479","Type":"ContainerStarted","Data":"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e"} Oct 30 22:21:18.737692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.737487 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:18.964353 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:18.964297 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" podStartSLOduration=2.118373823 podStartE2EDuration="5.964282869s" totalImagesPullingTime="3.845909046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:18.911270104 +0000 UTC m=+1606.880089669" watchObservedRunningTime="2026-10-30 22:21:18.964282869 +0000 UTC m=+1606.933102434" Oct 30 22:21:19.749655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:19.749621 4292 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lb4wk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:19.750192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:19.749686 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:22.749613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:22.749538 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:21:26.641177 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:21:26.641137 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:21:31.059884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:31.059849 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5"] Oct 30 22:21:31.110104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:31.110071 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7466096-mpnq5"] Oct 30 22:21:32.156064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:32.156027 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf441fe2-cde1-4a3b-b372-038ae1cb9058" path="/var/lib/kubelet/pods/cf441fe2-cde1-4a3b-b372-038ae1cb9058/volumes" Oct 30 22:21:35.796653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:35.796618 4292 scope.go:122] "RemoveContainer" containerID="5445f5f41c73d0ea6d3472aa44452218e145acd90d21c68f156faa60d3d8d7b3" Oct 30 22:21:35.876117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:35.876083 4292 scope.go:122] "RemoveContainer" containerID="371a4f7ac8fcd83aae7d7333ce7e5a5f377461410206208e7e4c9edee9f6aaa4" Oct 30 22:21:36.111648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:36.111281 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c"] Oct 30 22:21:36.395530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:36.390168 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e5677b-pth7c"] Oct 30 22:21:38.149874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:38.149837 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ab0f80-899c-4bf7-b001-3d17ce7d3ff6" path="/var/lib/kubelet/pods/60ab0f80-899c-4bf7-b001-3d17ce7d3ff6/volumes" Oct 30 22:21:47.072037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:47.072001 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x"] Oct 30 22:21:47.121607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:47.121565 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c6a71-x6f2x"] Oct 30 22:21:48.149368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:48.149336 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7edbe1-5f79-420a-923f-ee32ca22f1c4" path="/var/lib/kubelet/pods/9c7edbe1-5f79-420a-923f-ee32ca22f1c4/volumes" Oct 30 22:21:50.074564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.074517 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27"] Oct 30 22:21:50.129661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.129566 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e666106-smm27"] Oct 30 22:21:50.141458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.141420 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:21:50.161449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.161409 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.174621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.173382 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf58b6d-d909-453d-985b-e81c6bada8cc" path="/var/lib/kubelet/pods/ecf58b6d-d909-453d-985b-e81c6bada8cc/volumes" Oct 30 22:21:50.214566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.214533 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:21:50.220371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.219741 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-367f4491\"" Oct 30 22:21:50.295240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.294676 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-367f4491-dockercfg-x2sfw\"" Oct 30 22:21:50.296944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.296907 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ss4k\" (UniqueName: \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.297098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.297013 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.398274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.398196 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ss4k\" (UniqueName: \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.398421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.398334 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.399118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.399092 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.461345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.461317 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ss4k\" (UniqueName: \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\") pod \"payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:50.534629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:50.530080 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:51.174806 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:21:51.174775 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1447dc5_b8a2_4d1c_8283_e8951c7bfe87.slice/crio-c41190bef500eb4c2ae4075e420f9d72579a6302bd56b55ccfae8861201a98c5 WatchSource:0}: Error finding container c41190bef500eb4c2ae4075e420f9d72579a6302bd56b55ccfae8861201a98c5: Status 404 returned error can't find the container with id c41190bef500eb4c2ae4075e420f9d72579a6302bd56b55ccfae8861201a98c5 Oct 30 22:21:51.177387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:51.177360 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:21:51.237309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:51.237280 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" event={"ID":"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87","Type":"ContainerStarted","Data":"c41190bef500eb4c2ae4075e420f9d72579a6302bd56b55ccfae8861201a98c5"} Oct 30 22:21:52.247861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:52.247820 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" event={"ID":"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87","Type":"ContainerStarted","Data":"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a"} Oct 30 22:21:52.248236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:52.247921 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:21:52.369066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:21:52.369017 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" podStartSLOduration=2.369004241 podStartE2EDuration="2.369004241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:52.366850744 +0000 UTC m=+1640.335670310" watchObservedRunningTime="2026-10-30 22:21:52.369004241 +0000 UTC m=+1640.337823804" Oct 30 22:21:56.641189 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:21:56.641149 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:22:03.066040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:03.066000 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw"] Oct 30 22:22:03.112391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:03.112360 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5ecf97-9dpdw"] Oct 30 22:22:03.259834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:03.259804 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:22:04.086695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:04.086663 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j"] Oct 30 22:22:04.091543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:04.091519 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e23e2f-c9r4j"] Oct 30 22:22:04.155727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:04.155696 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd0b5a0-6be5-4675-ba87-a836e7e15966" path="/var/lib/kubelet/pods/3bd0b5a0-6be5-4675-ba87-a836e7e15966/volumes" Oct 30 22:22:04.156166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:04.156152 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8" path="/var/lib/kubelet/pods/91e1b1ba-2a4c-4eb3-a8cc-0254041f03e8/volumes" Oct 30 22:22:05.069529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:05.069490 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt"] Oct 30 22:22:05.117845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:05.117809 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53093ac1-nkrrt"] Oct 30 22:22:06.148391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:06.148356 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426c23ee-2a52-4b38-9c92-817115d5674a" path="/var/lib/kubelet/pods/426c23ee-2a52-4b38-9c92-817115d5674a/volumes" Oct 30 22:22:07.065609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:07.065556 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t"] Oct 30 22:22:07.117072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:07.117036 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0844ac23-4vk4t"] Oct 30 22:22:08.150582 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:08.150534 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc24a706-f079-4dce-97e5-95e7822ed722" path="/var/lib/kubelet/pods/cc24a706-f079-4dce-97e5-95e7822ed722/volumes" Oct 30 22:22:10.062159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:10.062119 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks"] Oct 30 22:22:10.152473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:10.152437 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b971b2-74wks"] Oct 30 22:22:12.152783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:12.152747 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8357d86-de78-4480-8d71-f6a3b9bdb81f" path="/var/lib/kubelet/pods/e8357d86-de78-4480-8d71-f6a3b9bdb81f/volumes" Oct 30 22:22:14.063762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:14.063722 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7"] Oct 30 22:22:14.113884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:14.113842 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75dd5f5c-5pql7"] Oct 30 22:22:14.150974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:14.150929 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79af5c0b-6de5-48fd-9190-5787e32c1971" path="/var/lib/kubelet/pods/79af5c0b-6de5-48fd-9190-5787e32c1971/volumes" Oct 30 22:22:14.519821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:14.519722 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:22:14.519821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:14.519796 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:17.132945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.132902 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": read tcp 10.134.0.2:41534->10.134.0.142:15021: read: connection reset by peer" start-of-body= Oct 30 22:22:17.133382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.132982 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": read tcp 10.134.0.2:41534->10.134.0.142:15021: read: connection reset by peer" Oct 30 22:22:17.595495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.595464 4292 generic.go:356] "Generic (PLEG): container finished" podID="b1df68f5-165d-4876-9379-27b2e703e508" containerID="fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6" exitCode=0 Oct 30 22:22:17.595706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.595537 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerDied","Data":"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6"} Oct 30 22:22:17.596075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.596034 4292 scope.go:122] "RemoveContainer" containerID="fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6" Oct 30 22:22:17.597994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.597962 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:17.598097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.598044 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:17.598097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.598092 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:17.598195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.598136 4292 generic.go:356] "Generic (PLEG): container finished" podID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerID="75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777" exitCode=0 Oct 30 22:22:17.598251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.598186 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerDied","Data":"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777"} Oct 30 22:22:17.598682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.598645 4292 scope.go:122] "RemoveContainer" containerID="75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777" Oct 30 22:22:17.600261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.600236 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:17.600345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.600317 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:17.600384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:17.600354 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:18.111276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:18.111229 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4"] Oct 30 22:22:18.118345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:18.118312 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88447a0b-hzms4"] Oct 30 22:22:18.152451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:18.152416 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c2d253-b8fc-4635-94d7-088452af5b03" path="/var/lib/kubelet/pods/e2c2d253-b8fc-4635-94d7-088452af5b03/volumes" Oct 30 22:22:18.612431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:18.612402 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerStarted","Data":"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83"} Oct 30 22:22:18.614505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:18.614479 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerStarted","Data":"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57"} Oct 30 22:22:19.153416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.153376 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:22:19.153999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.153972 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:19.154104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.154041 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:19.506658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.502002 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:22:19.506658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.502153 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:19.506658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:19.502196 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:20.154045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:20.154006 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:20.154526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:20.154081 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:20.484293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:20.484213 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:20.484293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:20.484265 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:21.153165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:21.153129 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.153354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:21.153178 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:21.492658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:21.488694 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.492658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:21.488773 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:22.156628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:22.154943 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.156628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:22.155017 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:22.484255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:22.484177 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.484255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:22.484234 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:23.058648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.058613 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn"] Oct 30 22:22:23.135150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.135116 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2812b60-nnhrn"] Oct 30 22:22:23.153916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.153888 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:23.154067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.153945 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:23.508117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.508017 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:23.508395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:23.508363 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:24.059785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.059754 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r"] Oct 30 22:22:24.131599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.130664 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d536d07-6rq8r"] Oct 30 22:22:24.150292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.150258 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6923b1bc-46ba-43d1-8f12-d011884089a0" path="/var/lib/kubelet/pods/6923b1bc-46ba-43d1-8f12-d011884089a0/volumes" Oct 30 22:22:24.150983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.150955 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70582417-c421-44cf-bd1d-284ccda74916" path="/var/lib/kubelet/pods/70582417-c421-44cf-bd1d-284ccda74916/volumes" Oct 30 22:22:24.153186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.153165 4292 patch_prober.go:29] interesting pod/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:22:24.153331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.153305 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:22:24.484533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.484459 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:22:24.484533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:24.484511 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:22:25.157475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.157446 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:22:25.157928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.157909 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:22:25.158515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.158496 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:22:25.493684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.490535 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:22:25.493684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.491931 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:22:25.493684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:25.492858 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:22:26.652805 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:22:26.652760 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:22:27.060144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:27.060104 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd"] Oct 30 22:22:27.110621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:27.110584 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62702043-8txmd"] Oct 30 22:22:28.059056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.059026 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j"] Oct 30 22:22:28.109118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.109075 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv"] Oct 30 22:22:28.113442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.113411 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6fb3b8-2qf2j"] Oct 30 22:22:28.117299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.117276 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a89c8a-8l9wv"] Oct 30 22:22:28.148872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.148845 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c973869-5d74-4b8e-9a37-8226029ed950" path="/var/lib/kubelet/pods/0c973869-5d74-4b8e-9a37-8226029ed950/volumes" Oct 30 22:22:28.149331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.149316 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee398022-de46-4098-8027-6af20f8d0ef0" path="/var/lib/kubelet/pods/ee398022-de46-4098-8027-6af20f8d0ef0/volumes" Oct 30 22:22:28.149773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:28.149759 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09fd51b-2e4a-403d-8a67-ebc8836f3fc5" path="/var/lib/kubelet/pods/f09fd51b-2e4a-403d-8a67-ebc8836f3fc5/volumes" Oct 30 22:22:34.463318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.463283 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk"] Oct 30 22:22:34.469731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.469700 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.514003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.513970 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.514222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.514194 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j99n\" (UniqueName: \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.514301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.514250 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.517621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.517598 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk"] Oct 30 22:22:34.562734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.562708 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:34.562884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.562856 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:22:34.562942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.562907 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:34.574096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.574078 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:34.616076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.616043 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.616255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.616235 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j99n\" (UniqueName: \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.616331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.616289 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.616970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.616940 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.618393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.618372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.679880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.679857 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j99n\" (UniqueName: \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\") pod \"maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:34.784207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:34.784142 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:35.163061 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:22:35.163021 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd85bc7_bfa7_4bba_8659_d553ba579cb5.slice/crio-c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5 WatchSource:0}: Error finding container c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5: Status 404 returned error can't find the container with id c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5 Oct 30 22:22:35.164635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:35.164605 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk"] Oct 30 22:22:35.797692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:35.797650 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" event={"ID":"bcd85bc7-bfa7-4bba-8659-d553ba579cb5","Type":"ContainerStarted","Data":"47d27a6fb80e98ffa1791758e6899c45e6917a43f1ac466f2f86fdd6af784064"} Oct 30 22:22:35.798125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:35.797716 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" event={"ID":"bcd85bc7-bfa7-4bba-8659-d553ba579cb5","Type":"ContainerStarted","Data":"c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5"} Oct 30 22:22:35.918291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:35.918236 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" podStartSLOduration=1.9182228380000002 podStartE2EDuration="1.918222838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:35.917199393 +0000 UTC m=+1683.886018957" watchObservedRunningTime="2026-10-30 22:22:35.918222838 +0000 UTC m=+1683.887042402" Oct 30 22:22:36.264624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.264593 4292 scope.go:122] "RemoveContainer" containerID="60132aa4a3291e616a428f227a321adfb39a9e0b4dffdb794a015754560b8808" Oct 30 22:22:36.275410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.275391 4292 scope.go:122] "RemoveContainer" containerID="192d52f32ae838409595ff5b5228c5eda288b3e0772e53fe82e85ba9d771d79c" Oct 30 22:22:36.286021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.285999 4292 scope.go:122] "RemoveContainer" containerID="6830d57de87b3e334c40fc28abbc3fc3297f62a55e7777f7f6307424ce9219e6" Oct 30 22:22:36.295194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.295177 4292 scope.go:122] "RemoveContainer" containerID="8a02a7daac6c4a7a648f47e7fa5203884e806e7b2f201de5acb089aa0085efd9" Oct 30 22:22:36.808849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.808812 4292 generic.go:356] "Generic (PLEG): container finished" podID="bcd85bc7-bfa7-4bba-8659-d553ba579cb5" containerID="47d27a6fb80e98ffa1791758e6899c45e6917a43f1ac466f2f86fdd6af784064" exitCode=0 Oct 30 22:22:36.809267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:36.808911 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" event={"ID":"bcd85bc7-bfa7-4bba-8659-d553ba579cb5","Type":"ContainerDied","Data":"47d27a6fb80e98ffa1791758e6899c45e6917a43f1ac466f2f86fdd6af784064"} Oct 30 22:22:37.063551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:37.063465 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg"] Oct 30 22:22:37.071094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:37.071062 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c393eab2-zmtwg"] Oct 30 22:22:37.992774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:37.992750 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:38.075993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.075924 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\") pod \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " Oct 30 22:22:38.076113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.076018 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\" (UniqueName: \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\") pod \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " Oct 30 22:22:38.076219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.076203 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\") pod \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\" (UID: \"bcd85bc7-bfa7-4bba-8659-d553ba579cb5\") " Oct 30 22:22:38.076517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.076487 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca" pod "bcd85bc7-bfa7-4bba-8659-d553ba579cb5" (UID: "bcd85bc7-bfa7-4bba-8659-d553ba579cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:38.076872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.076849 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.077941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.077918 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca" pod "bcd85bc7-bfa7-4bba-8659-d553ba579cb5" (UID: "bcd85bc7-bfa7-4bba-8659-d553ba579cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:38.078193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.078170 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n" pod "bcd85bc7-bfa7-4bba-8659-d553ba579cb5" (UID: "bcd85bc7-bfa7-4bba-8659-d553ba579cb5"). InnerVolumeSpecName "kube-api-access-4j99n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:38.150503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.150464 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014fd968-cad3-48b0-8622-0700478b90af" path="/var/lib/kubelet/pods/014fd968-cad3-48b0-8622-0700478b90af/volumes" Oct 30 22:22:38.179148 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.179119 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.179329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.179153 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j99n\" (UniqueName: \"kubernetes.io/projected/bcd85bc7-bfa7-4bba-8659-d553ba579cb5-kube-api-access-4j99n\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.830871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.830841 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" Oct 30 22:22:38.830871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.830854 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk" event={"ID":"bcd85bc7-bfa7-4bba-8659-d553ba579cb5","Type":"ContainerDied","Data":"c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5"} Oct 30 22:22:38.831090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:38.830906 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4783f8278026048d82b2cd5b84852107d674e3650e7435a83d16de3fbca0fe5" Oct 30 22:22:41.058645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:41.058598 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt"] Oct 30 22:22:41.108506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:41.108473 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a535060-qqhxt"] Oct 30 22:22:42.058850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:42.058813 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg"] Oct 30 22:22:42.109330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:42.109298 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235bf2b5-npljg"] Oct 30 22:22:42.148903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:42.148870 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d6b4ca-3688-49db-acbc-02a111d043cc" path="/var/lib/kubelet/pods/29d6b4ca-3688-49db-acbc-02a111d043cc/volumes" Oct 30 22:22:42.149331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:42.149316 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b3da33-955a-45c9-862b-a6c70b8879cb" path="/var/lib/kubelet/pods/e5b3da33-955a-45c9-862b-a6c70b8879cb/volumes" Oct 30 22:22:45.303430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:45.303397 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:22:45.432287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:45.432257 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:22:48.060250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:48.060219 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685"] Oct 30 22:22:48.108681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:48.108646 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec14fab-gm685"] Oct 30 22:22:48.148947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:48.148917 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2289fad4-dd54-4407-9385-ff14d5b19faf" path="/var/lib/kubelet/pods/2289fad4-dd54-4407-9385-ff14d5b19faf/volumes" Oct 30 22:22:50.304122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.304084 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" podUID="3991f2e3-1c48-416f-a37e-d814d9c64381" containerName="payload-processing" containerID="cri-o://2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf" gracePeriod=25 Oct 30 22:22:50.433433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.433394 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" podUID="bf923105-2634-4a7c-93f4-a2efef0dd57f" containerName="payload-pre-processing" containerID="cri-o://10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de" gracePeriod=25 Oct 30 22:22:50.604792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.604766 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:22:50.681403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.681356 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\" (UniqueName: \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\") pod \"3991f2e3-1c48-416f-a37e-d814d9c64381\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " Oct 30 22:22:50.681626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.681482 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\") pod \"3991f2e3-1c48-416f-a37e-d814d9c64381\" (UID: \"3991f2e3-1c48-416f-a37e-d814d9c64381\") " Oct 30 22:22:50.681929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.681900 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume" pod "3991f2e3-1c48-416f-a37e-d814d9c64381" (UID: "3991f2e3-1c48-416f-a37e-d814d9c64381"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:50.682517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.682492 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3991f2e3-1c48-416f-a37e-d814d9c64381-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.683560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.683537 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5" pod "3991f2e3-1c48-416f-a37e-d814d9c64381" (UID: "3991f2e3-1c48-416f-a37e-d814d9c64381"). InnerVolumeSpecName "kube-api-access-7b4t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:50.717859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.717823 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:22:50.783383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.783344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\") pod \"bf923105-2634-4a7c-93f4-a2efef0dd57f\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " Oct 30 22:22:50.783555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.783420 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\" (UniqueName: \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\") pod \"bf923105-2634-4a7c-93f4-a2efef0dd57f\" (UID: \"bf923105-2634-4a7c-93f4-a2efef0dd57f\") " Oct 30 22:22:50.783821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.783794 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume" pod "bf923105-2634-4a7c-93f4-a2efef0dd57f" (UID: "bf923105-2634-4a7c-93f4-a2efef0dd57f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:50.784129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.784103 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf923105-2634-4a7c-93f4-a2efef0dd57f-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.784254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.784131 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4t5\" (UniqueName: \"kubernetes.io/projected/3991f2e3-1c48-416f-a37e-d814d9c64381-kube-api-access-7b4t5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.786176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.786154 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf" pod "bf923105-2634-4a7c-93f4-a2efef0dd57f" (UID: "bf923105-2634-4a7c-93f4-a2efef0dd57f"). InnerVolumeSpecName "kube-api-access-2d2tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:50.885623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.885531 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2tf\" (UniqueName: \"kubernetes.io/projected/bf923105-2634-4a7c-93f4-a2efef0dd57f-kube-api-access-2d2tf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.940253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.940220 4292 generic.go:356] "Generic (PLEG): container finished" podID="bf923105-2634-4a7c-93f4-a2efef0dd57f" containerID="10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de" exitCode=0 Oct 30 22:22:50.940424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.940273 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" Oct 30 22:22:50.940424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.940301 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" event={"ID":"bf923105-2634-4a7c-93f4-a2efef0dd57f","Type":"ContainerDied","Data":"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de"} Oct 30 22:22:50.940424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.940376 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5" event={"ID":"bf923105-2634-4a7c-93f4-a2efef0dd57f","Type":"ContainerDied","Data":"259fa56a443c2c978c8b00962aaa340ff0441f7eca950dac58d575cd1bc356a2"} Oct 30 22:22:50.940424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.940404 4292 scope.go:122] "RemoveContainer" containerID="10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de" Oct 30 22:22:50.941896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.941871 4292 generic.go:356] "Generic (PLEG): container finished" podID="3991f2e3-1c48-416f-a37e-d814d9c64381" containerID="2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf" exitCode=0 Oct 30 22:22:50.942019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.941910 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" Oct 30 22:22:50.942019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.941952 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" event={"ID":"3991f2e3-1c48-416f-a37e-d814d9c64381","Type":"ContainerDied","Data":"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf"} Oct 30 22:22:50.942019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.942001 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf" event={"ID":"3991f2e3-1c48-416f-a37e-d814d9c64381","Type":"ContainerDied","Data":"e988c31aa64bb5764dd9c97848d91b399bc2d0e00c437d01421235affe159363"} Oct 30 22:22:50.952166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.952148 4292 scope.go:122] "RemoveContainer" containerID="10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de" Oct 30 22:22:50.952475 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:22:50.952454 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de\": container with ID starting with 10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de not found: ID does not exist" containerID="10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de" Oct 30 22:22:50.952540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.952481 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de"} err="failed to get container status \"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de\": rpc error: code = NotFound desc = could not find container \"10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de\": container with ID starting with 10488543f1cad1b5da6499704f24b5ba19817a2274bc0952ae39b8fcfd9092de not found: ID does not exist" Oct 30 22:22:50.952540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.952498 4292 scope.go:122] "RemoveContainer" containerID="2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf" Oct 30 22:22:50.962113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.962085 4292 scope.go:122] "RemoveContainer" containerID="2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf" Oct 30 22:22:50.962369 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:22:50.962349 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf\": container with ID starting with 2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf not found: ID does not exist" containerID="2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf" Oct 30 22:22:50.962430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:50.962378 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf"} err="failed to get container status \"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf\": rpc error: code = NotFound desc = could not find container \"2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf\": container with ID starting with 2fe1ccc9d2ce6779e69e649733661cab8d86847febba59349c4f1bfa2b1e2caf not found: ID does not exist" Oct 30 22:22:51.127737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.127705 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:22:51.132919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.132900 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-05eb684c-54d858cfd7-55ttf"] Oct 30 22:22:51.242229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.242162 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:22:51.248624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.248599 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-05eb684c-77d94b7679-xv5x5"] Oct 30 22:22:51.357221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.357184 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:22:51.357816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.357432 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" containerID="cri-o://66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da" gracePeriod=30 Oct 30 22:22:51.955243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.955211 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:22:51.955963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.955949 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bcd85bc7-bfa7-4bba-8659-d553ba579cb5" containerName="revoke-keys" Oct 30 22:22:51.956014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.955965 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd85bc7-bfa7-4bba-8659-d553ba579cb5" containerName="revoke-keys" Oct 30 22:22:51.956014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.955986 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bf923105-2634-4a7c-93f4-a2efef0dd57f" containerName="payload-pre-processing" Oct 30 22:22:51.956014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.955992 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf923105-2634-4a7c-93f4-a2efef0dd57f" containerName="payload-pre-processing" Oct 30 22:22:51.956116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.956028 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3991f2e3-1c48-416f-a37e-d814d9c64381" containerName="payload-processing" Oct 30 22:22:51.956116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.956033 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3991f2e3-1c48-416f-a37e-d814d9c64381" containerName="payload-processing" Oct 30 22:22:51.956194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.956129 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd85bc7-bfa7-4bba-8659-d553ba579cb5" containerName="revoke-keys" Oct 30 22:22:51.956194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.956141 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf923105-2634-4a7c-93f4-a2efef0dd57f" containerName="payload-pre-processing" Oct 30 22:22:51.956194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.956155 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3991f2e3-1c48-416f-a37e-d814d9c64381" containerName="payload-processing" Oct 30 22:22:51.959395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:51.959376 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.015852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.015817 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r29v\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.015994 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016177 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016221 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016343 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016434 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016524 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016631 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.016710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.016685 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.024675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.024649 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:22:52.067150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.067120 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18515933-openshift-default-dockercfg-fpd9x\"" Oct 30 22:22:52.086642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.086613 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8"] Oct 30 22:22:52.091632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.091601 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e32564-twvg8"] Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.119675 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.119793 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.119867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.119899 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.119941 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r29v\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120142 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120280 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120308 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120668 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120753 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.120277 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.121154 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.121372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.123121 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.124667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.123869 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.148746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.148718 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3991f2e3-1c48-416f-a37e-d814d9c64381" path="/var/lib/kubelet/pods/3991f2e3-1c48-416f-a37e-d814d9c64381/volumes" Oct 30 22:22:52.149400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.149373 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ea30cc-3190-46c7-9c6b-d1eeb8054ba5" path="/var/lib/kubelet/pods/61ea30cc-3190-46c7-9c6b-d1eeb8054ba5/volumes" Oct 30 22:22:52.149987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.149965 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf923105-2634-4a7c-93f4-a2efef0dd57f" path="/var/lib/kubelet/pods/bf923105-2634-4a7c-93f4-a2efef0dd57f/volumes" Oct 30 22:22:52.184508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.184476 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r29v\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.185161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.185128 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\") pod \"e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.273614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.273508 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:52.735366 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:22:52.735330 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb7cda7_c269_461a_ac65_2c400eec3374.slice/crio-6b24124e9c8f41cbd2507d25fab15f996142dc061a05441a6a1ec298cdd8a82e WatchSource:0}: Error finding container 6b24124e9c8f41cbd2507d25fab15f996142dc061a05441a6a1ec298cdd8a82e: Status 404 returned error can't find the container with id 6b24124e9c8f41cbd2507d25fab15f996142dc061a05441a6a1ec298cdd8a82e Oct 30 22:22:52.735795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.735474 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:22:52.738044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.738012 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:52.738171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.738071 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:52.738171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.738113 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:22:52.966659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.966624 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerStarted","Data":"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f"} Oct 30 22:22:52.966818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:52.966687 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerStarted","Data":"6b24124e9c8f41cbd2507d25fab15f996142dc061a05441a6a1ec298cdd8a82e"} Oct 30 22:22:53.060560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.060518 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp"] Oct 30 22:22:53.075549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.075520 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ae279e-k6jzp"] Oct 30 22:22:53.108989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.108940 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podStartSLOduration=2.108928275 podStartE2EDuration="2.108928275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:53.092705479 +0000 UTC m=+1701.061525043" watchObservedRunningTime="2026-10-30 22:22:53.108928275 +0000 UTC m=+1701.077747838" Oct 30 22:22:53.274273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.274239 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:53.280266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.280236 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:53.974428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.974392 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:53.975302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:53.975283 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:22:54.153696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:54.153657 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdfaac1-06d2-4602-8967-a0b7c45c76fd" path="/var/lib/kubelet/pods/ebdfaac1-06d2-4602-8967-a0b7c45c76fd/volumes" Oct 30 22:22:56.641642 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:22:56.641604 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:22:56.648799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.648777 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:22:56.822937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.822896 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823007 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823056 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823207 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823335 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823388 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823436 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823542 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823499 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823614 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\") pod \"0b68b888-0376-4a8f-8c03-c984fad1f205\" (UID: \"0b68b888-0376-4a8f-8c03-c984fad1f205\") " Oct 30 22:22:56.823751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823390 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:56.823871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823478 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:56.823871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.823657 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:56.824083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824058 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:56.824283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824249 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:56.824703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824671 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.824703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824700 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.824863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824715 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.824863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824730 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b68b888-0376-4a8f-8c03-c984fad1f205-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.824863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.824744 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.826071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.825951 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:56.826170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.826123 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:56.826650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.826615 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:56.827278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.827259 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz" pod "0b68b888-0376-4a8f-8c03-c984fad1f205" (UID: "0b68b888-0376-4a8f-8c03-c984fad1f205"). InnerVolumeSpecName "kube-api-access-g5ghz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:56.926115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.926083 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.926115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.926115 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ghz\" (UniqueName: \"kubernetes.io/projected/0b68b888-0376-4a8f-8c03-c984fad1f205-kube-api-access-g5ghz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.926345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.926128 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:56.926345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:56.926140 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b68b888-0376-4a8f-8c03-c984fad1f205-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:57.004894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.004853 4292 generic.go:356] "Generic (PLEG): container finished" podID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerID="66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da" exitCode=0 Oct 30 22:22:57.005065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.004898 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" Oct 30 22:22:57.005065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.004923 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" event={"ID":"0b68b888-0376-4a8f-8c03-c984fad1f205","Type":"ContainerDied","Data":"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da"} Oct 30 22:22:57.005065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.004973 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8" event={"ID":"0b68b888-0376-4a8f-8c03-c984fad1f205","Type":"ContainerDied","Data":"a4f66fcbc5e60d62e93920e1ba89883bc209beacca8991cef32fb806aa822236"} Oct 30 22:22:57.005065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.004993 4292 scope.go:122] "RemoveContainer" containerID="66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da" Oct 30 22:22:57.018078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.018060 4292 scope.go:122] "RemoveContainer" containerID="66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da" Oct 30 22:22:57.018365 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:22:57.018339 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da\": container with ID starting with 66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da not found: ID does not exist" containerID="66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da" Oct 30 22:22:57.018448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.018376 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da"} err="failed to get container status \"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da\": rpc error: code = NotFound desc = could not find container \"66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da\": container with ID starting with 66c4ac30f5deec4de3e3dbb7d2ad1a23a71ebb091fa7b20d4776dce2b1a907da not found: ID does not exist" Oct 30 22:22:57.073555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.073461 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb"] Oct 30 22:22:57.077750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.077718 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c333aa4-57nrb"] Oct 30 22:22:57.189760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.189728 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:22:57.196219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:57.196192 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-crhf8"] Oct 30 22:22:58.149741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:58.149711 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" path="/var/lib/kubelet/pods/0b68b888-0376-4a8f-8c03-c984fad1f205/volumes" Oct 30 22:22:58.150239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:58.150223 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73415ce-d3fc-43b0-a966-43007a4af66d" path="/var/lib/kubelet/pods/f73415ce-d3fc-43b0-a966-43007a4af66d/volumes" Oct 30 22:22:59.061083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:59.061048 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489"] Oct 30 22:22:59.110831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:22:59.110791 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5f5b6b-md489"] Oct 30 22:23:00.079417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.079379 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:23:00.080278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.080254 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" Oct 30 22:23:00.080278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.080277 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" Oct 30 22:23:00.080542 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.080397 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b68b888-0376-4a8f-8c03-c984fad1f205" containerName="istio-proxy" Oct 30 22:23:00.085747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.085727 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.090504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.090467 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.090690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.090508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnscf\" (UniqueName: \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.161068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.161029 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0fd961-9ba2-4869-8239-62a5bbc58cc7" path="/var/lib/kubelet/pods/ea0fd961-9ba2-4869-8239-62a5bbc58cc7/volumes" Oct 30 22:23:00.171806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.171770 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:23:00.193549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.193511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.195181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.195150 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnscf\" (UniqueName: \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.207128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.207098 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18515933\"" Oct 30 22:23:00.207526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.207505 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18515933-dockercfg-tzdn6\"" Oct 30 22:23:00.220639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.210625 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.260667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.257969 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:23:00.262742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.262704 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnscf\" (UniqueName: \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.268609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.268566 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.298595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.297225 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7m5r\" (UniqueName: \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.298595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.297405 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.298595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.297608 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:23:00.399445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.399361 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.399640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.399548 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7m5r\" (UniqueName: \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.400344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.400320 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.403619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.403554 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:00.468758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.465664 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7m5r\" (UniqueName: \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.670476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.670384 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:00.808732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:00.808057 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:23:00.815620 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:00.813051 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc306715_040b_43b3_84c7_4c3eb267d5a0.slice/crio-63c44281755f6eea51069ae45bed9e8de9c92f234404b8b759020de95a95fe10 WatchSource:0}: Error finding container 63c44281755f6eea51069ae45bed9e8de9c92f234404b8b759020de95a95fe10: Status 404 returned error can't find the container with id 63c44281755f6eea51069ae45bed9e8de9c92f234404b8b759020de95a95fe10 Oct 30 22:23:01.041926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:01.041846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" event={"ID":"dc306715-040b-43b3-84c7-4c3eb267d5a0","Type":"ContainerStarted","Data":"028a8f55a2896c67004897e95140bac9d7089525d10c5dd490122b784d25b78a"} Oct 30 22:23:01.041926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:01.041904 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" event={"ID":"dc306715-040b-43b3-84c7-4c3eb267d5a0","Type":"ContainerStarted","Data":"63c44281755f6eea51069ae45bed9e8de9c92f234404b8b759020de95a95fe10"} Oct 30 22:23:01.042141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:01.042125 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:01.097873 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:01.097842 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a986428_4dd2_4f80_ad35_e553b64c2d3e.slice/crio-43c219e6f3b5022227d6b4bf283ec3f3f6c8182d94b2bb33a50d5f4acfade445 WatchSource:0}: Error finding container 43c219e6f3b5022227d6b4bf283ec3f3f6c8182d94b2bb33a50d5f4acfade445: Status 404 returned error can't find the container with id 43c219e6f3b5022227d6b4bf283ec3f3f6c8182d94b2bb33a50d5f4acfade445 Oct 30 22:23:01.099454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:01.099430 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:23:01.162785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:01.162743 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" podStartSLOduration=1.162728767 podStartE2EDuration="1.162728767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:01.161478824 +0000 UTC m=+1709.130298411" watchObservedRunningTime="2026-10-30 22:23:01.162728767 +0000 UTC m=+1709.131548331" Oct 30 22:23:02.052055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.052022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" event={"ID":"6a986428-4dd2-4f80-ad35-e553b64c2d3e","Type":"ContainerStarted","Data":"95873856803e048c64e9c95fca879a785a8157a1a781798a9bd8e3faf9a694c3"} Oct 30 22:23:02.052266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.052075 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" event={"ID":"6a986428-4dd2-4f80-ad35-e553b64c2d3e","Type":"ContainerStarted","Data":"43c219e6f3b5022227d6b4bf283ec3f3f6c8182d94b2bb33a50d5f4acfade445"} Oct 30 22:23:02.052266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.052150 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:02.175288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.175230 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" podStartSLOduration=2.175212906 podStartE2EDuration="2.175212906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:02.171613031 +0000 UTC m=+1710.140432595" watchObservedRunningTime="2026-10-30 22:23:02.175212906 +0000 UTC m=+1710.144032473" Oct 30 22:23:02.591980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.591946 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn"] Oct 30 22:23:02.596511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.596489 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.651820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.651796 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:02.651967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.651953 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:23:02.652083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.652072 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:02.652325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.652305 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:02.680009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.679982 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sxz\" (UniqueName: \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.680220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.680199 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.680327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.680308 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.707161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.707130 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn"] Oct 30 22:23:02.782004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.781978 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sxz\" (UniqueName: \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.782128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.782101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.782171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.782146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.782705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.782688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.784451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.784432 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.845093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.845035 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sxz\" (UniqueName: \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\") pod \"maas-api-revoke-keys-e2e-derive-678b1628-jhzkn\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:02.911804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:02.911779 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:03.082433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:03.082397 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr"] Oct 30 22:23:03.089410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:03.089359 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100d9e93-7wpwr"] Oct 30 22:23:03.309861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:03.309831 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn"] Oct 30 22:23:03.310210 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:03.309981 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6e8785_0a0c_4072_8033_4690e2148780.slice/crio-c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7 WatchSource:0}: Error finding container c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7: Status 404 returned error can't find the container with id c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7 Oct 30 22:23:03.991007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:03.990931 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc"] Oct 30 22:23:03.995399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:03.995379 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.057266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.057229 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc"] Oct 30 22:23:04.094560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.094528 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" event={"ID":"2b6e8785-0a0c-4072-8033-4690e2148780","Type":"ContainerStarted","Data":"660ffaa2393a1fd303036766f12eabd600da1103db8a3ee3c64077b97744d7ea"} Oct 30 22:23:04.094719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.094618 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" event={"ID":"2b6e8785-0a0c-4072-8033-4690e2148780","Type":"ContainerStarted","Data":"c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7"} Oct 30 22:23:04.148492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.148460 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656d48c0-69db-4eba-a11f-267f4344aedf" path="/var/lib/kubelet/pods/656d48c0-69db-4eba-a11f-267f4344aedf/volumes" Oct 30 22:23:04.156738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.156713 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.156832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.156745 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.156899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.156852 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgkv\" (UniqueName: \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.239470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.239415 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" podStartSLOduration=2.239398177 podStartE2EDuration="2.239398177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:04.236770459 +0000 UTC m=+1712.205590050" watchObservedRunningTime="2026-10-30 22:23:04.239398177 +0000 UTC m=+1712.208217743" Oct 30 22:23:04.258928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.258799 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgkv\" (UniqueName: \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.259379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.259335 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.259379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.259378 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.260018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.259995 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.262032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.262008 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.322371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.322342 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgkv\" (UniqueName: \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\") pod \"maas-api-revoke-sub-0140d87b-7q2nc\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.610929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.610895 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:04.998718 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:04.998691 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec8624e_d4fb_46d7_9a2a_a5ddc15f8328.slice/crio-84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b WatchSource:0}: Error finding container 84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b: Status 404 returned error can't find the container with id 84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b Oct 30 22:23:04.999031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:04.999008 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc"] Oct 30 22:23:05.104036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:05.104005 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" event={"ID":"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328","Type":"ContainerStarted","Data":"84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b"} Oct 30 22:23:05.105617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:05.105557 4292 generic.go:356] "Generic (PLEG): container finished" podID="2b6e8785-0a0c-4072-8033-4690e2148780" containerID="660ffaa2393a1fd303036766f12eabd600da1103db8a3ee3c64077b97744d7ea" exitCode=0 Oct 30 22:23:05.105723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:05.105642 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" event={"ID":"2b6e8785-0a0c-4072-8033-4690e2148780","Type":"ContainerDied","Data":"660ffaa2393a1fd303036766f12eabd600da1103db8a3ee3c64077b97744d7ea"} Oct 30 22:23:06.116238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.116203 4292 generic.go:356] "Generic (PLEG): container finished" podID="3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" containerID="74a7c135ac2e50c97dcf1701fd2d68690923f9f6650e130535cb82c5f1dd037b" exitCode=0 Oct 30 22:23:06.116642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.116279 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" event={"ID":"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328","Type":"ContainerDied","Data":"74a7c135ac2e50c97dcf1701fd2d68690923f9f6650e130535cb82c5f1dd037b"} Oct 30 22:23:06.287850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.287827 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:06.390188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.390157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\") pod \"2b6e8785-0a0c-4072-8033-4690e2148780\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " Oct 30 22:23:06.390340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.390292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\") pod \"2b6e8785-0a0c-4072-8033-4690e2148780\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " Oct 30 22:23:06.390541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.390519 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca" pod "2b6e8785-0a0c-4072-8033-4690e2148780" (UID: "2b6e8785-0a0c-4072-8033-4690e2148780"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:06.390612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.390529 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\" (UniqueName: \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\") pod \"2b6e8785-0a0c-4072-8033-4690e2148780\" (UID: \"2b6e8785-0a0c-4072-8033-4690e2148780\") " Oct 30 22:23:06.391227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.391206 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:06.392343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.392315 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca" pod "2b6e8785-0a0c-4072-8033-4690e2148780" (UID: "2b6e8785-0a0c-4072-8033-4690e2148780"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:06.392556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.392532 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz" pod "2b6e8785-0a0c-4072-8033-4690e2148780" (UID: "2b6e8785-0a0c-4072-8033-4690e2148780"). InnerVolumeSpecName "kube-api-access-t9sxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:06.492409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.492369 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6e8785-0a0c-4072-8033-4690e2148780-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:06.492409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:06.492406 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9sxz\" (UniqueName: \"kubernetes.io/projected/2b6e8785-0a0c-4072-8033-4690e2148780-kube-api-access-t9sxz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:07.126160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.126123 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" event={"ID":"2b6e8785-0a0c-4072-8033-4690e2148780","Type":"ContainerDied","Data":"c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7"} Oct 30 22:23:07.126586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.126181 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c110a30e3e837694fa35b71791d15d9865260aa2624a073437ed3b9010534dc7" Oct 30 22:23:07.126586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.126140 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn" Oct 30 22:23:07.284852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.284822 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:07.303975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.303946 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\" (UniqueName: \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\") pod \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " Oct 30 22:23:07.304095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.304056 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\") pod \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " Oct 30 22:23:07.304236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.304222 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\") pod \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\" (UID: \"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328\") " Oct 30 22:23:07.304627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.304603 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca" pod "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" (UID: "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:07.304798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.304783 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:07.305991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.305965 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca" pod "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" (UID: "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:07.306053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.305989 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv" pod "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" (UID: "3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328"). InnerVolumeSpecName "kube-api-access-qrgkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:07.406656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.406563 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgkv\" (UniqueName: \"kubernetes.io/projected/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-kube-api-access-qrgkv\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:07.406656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:07.406621 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:08.136461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:08.136428 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" event={"ID":"3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328","Type":"ContainerDied","Data":"84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b"} Oct 30 22:23:08.136461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:08.136459 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc" Oct 30 22:23:08.136919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:08.136478 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84df953f7115c70f386a80d8d5f8d0ac314dde3e50694f05844562b0c0689f8b" Oct 30 22:23:12.055862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:12.055829 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:23:13.082388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:13.082358 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:23:16.120489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:16.120451 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:23:16.239782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:16.239748 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:23:21.121305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.121264 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" podUID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" containerName="payload-processing" containerID="cri-o://66e2687f7372d820747fc4dd4eaeed1ee00390de823130dfe49246d2558cc3d5" gracePeriod=25 Oct 30 22:23:21.240185 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.240117 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" podUID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" containerName="payload-pre-processing" containerID="cri-o://a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b" gracePeriod=25 Oct 30 22:23:21.256998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.256965 4292 generic.go:356] "Generic (PLEG): container finished" podID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" containerID="66e2687f7372d820747fc4dd4eaeed1ee00390de823130dfe49246d2558cc3d5" exitCode=0 Oct 30 22:23:21.257358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.257271 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" event={"ID":"5a80a521-b11f-4fe0-a73c-c7aaffed564d","Type":"ContainerDied","Data":"66e2687f7372d820747fc4dd4eaeed1ee00390de823130dfe49246d2558cc3d5"} Oct 30 22:23:21.464254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.464226 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:23:21.607310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.607276 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:23:21.622486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.622457 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\" (UniqueName: \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\") pod \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " Oct 30 22:23:21.622808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.622788 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\") pod \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\" (UID: \"5a80a521-b11f-4fe0-a73c-c7aaffed564d\") " Oct 30 22:23:21.623194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.623171 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume" pod "5a80a521-b11f-4fe0-a73c-c7aaffed564d" (UID: "5a80a521-b11f-4fe0-a73c-c7aaffed564d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:21.623428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.623406 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a80a521-b11f-4fe0-a73c-c7aaffed564d-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:21.624759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.624738 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt" pod "5a80a521-b11f-4fe0-a73c-c7aaffed564d" (UID: "5a80a521-b11f-4fe0-a73c-c7aaffed564d"). InnerVolumeSpecName "kube-api-access-frvnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:21.724556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.724472 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\") pod \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " Oct 30 22:23:21.724741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.724557 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\" (UniqueName: \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\") pod \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\" (UID: \"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2\") " Oct 30 22:23:21.724932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.724904 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume" pod "d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" (UID: "d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:21.725331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.725306 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvnt\" (UniqueName: \"kubernetes.io/projected/5a80a521-b11f-4fe0-a73c-c7aaffed564d-kube-api-access-frvnt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:21.725467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.725332 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:21.728957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.728928 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr" pod "d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" (UID: "d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2"). InnerVolumeSpecName "kube-api-access-flxrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:21.826951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:21.826911 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxrr\" (UniqueName: \"kubernetes.io/projected/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2-kube-api-access-flxrr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:22.061039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.061006 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp"] Oct 30 22:23:22.065402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.065372 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb8576-jlwjp"] Oct 30 22:23:22.148771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.148744 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc15dd84-b0d7-4190-93fe-995ece656ee3" path="/var/lib/kubelet/pods/fc15dd84-b0d7-4190-93fe-995ece656ee3/volumes" Oct 30 22:23:22.266963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.266919 4292 generic.go:356] "Generic (PLEG): container finished" podID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" containerID="a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b" exitCode=0 Oct 30 22:23:22.267122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.266973 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" Oct 30 22:23:22.267122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.266994 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" event={"ID":"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2","Type":"ContainerDied","Data":"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b"} Oct 30 22:23:22.267122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.267048 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf" event={"ID":"d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2","Type":"ContainerDied","Data":"ff0b53bfbe03d7bffee188ef7f24e44c719b31626718c5dd85f794e3d5897bf4"} Oct 30 22:23:22.267122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.267069 4292 scope.go:122] "RemoveContainer" containerID="a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b" Oct 30 22:23:22.268918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.268895 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" event={"ID":"5a80a521-b11f-4fe0-a73c-c7aaffed564d","Type":"ContainerDied","Data":"c31019d6193e8eefee3e23463b252e32732df9efa283c62ad3cff12a19b23ee5"} Oct 30 22:23:22.269020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.268993 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj" Oct 30 22:23:22.279480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.279463 4292 scope.go:122] "RemoveContainer" containerID="a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b" Oct 30 22:23:22.279766 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:22.279747 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b\": container with ID starting with a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b not found: ID does not exist" containerID="a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b" Oct 30 22:23:22.279848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.279773 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b"} err="failed to get container status \"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b\": rpc error: code = NotFound desc = could not find container \"a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b\": container with ID starting with a1918c60b98a2e74bcbb727a71a8273577aa60f306ed4f476380e6da3ea8836b not found: ID does not exist" Oct 30 22:23:22.279848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.279788 4292 scope.go:122] "RemoveContainer" containerID="66e2687f7372d820747fc4dd4eaeed1ee00390de823130dfe49246d2558cc3d5" Oct 30 22:23:22.452719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.452647 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:23:22.484114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.484087 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-678b1628-f8b5c8995-dntqj"] Oct 30 22:23:22.598641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.598613 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:23:22.607424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:22.607397 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-678b1628-6699bb9ff9-q4zvf"] Oct 30 22:23:24.149015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:24.148979 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" path="/var/lib/kubelet/pods/5a80a521-b11f-4fe0-a73c-c7aaffed564d/volumes" Oct 30 22:23:24.149453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:24.149437 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" path="/var/lib/kubelet/pods/d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2/volumes" Oct 30 22:23:26.641687 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:26.641651 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:23:36.343788 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.343757 4292 scope.go:122] "RemoveContainer" containerID="b2e7b38c8a61b4fe29107c5814189feaeaa049abd3b254f746de2d0b985c586d" Oct 30 22:23:36.354919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.354899 4292 scope.go:122] "RemoveContainer" containerID="288eb44504a0f218180b3f322db6e6d78f6c11c79308c24bf9fd20989c73ea48" Oct 30 22:23:36.364946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.364928 4292 scope.go:122] "RemoveContainer" containerID="2d46a72383804420ffacbd40ee0f70900ef591d85d4f063126bc22d721ffe753" Oct 30 22:23:36.376340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.376318 4292 scope.go:122] "RemoveContainer" containerID="ec77fdf9a04da799a4144c28d241776d1daee45a2893adbd8434b0c253200e51" Oct 30 22:23:36.400151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.393096 4292 scope.go:122] "RemoveContainer" containerID="ca8a609433add8c46baa452d83ad6d76da12f6121709ece54a79e8aa89cef306" Oct 30 22:23:36.411111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.411083 4292 scope.go:122] "RemoveContainer" containerID="2a6fbbc2d5c699301734b976cfb028fb4e6f8c00c1f6e7a3bfee8e9c066e84d0" Oct 30 22:23:36.421809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.421730 4292 scope.go:122] "RemoveContainer" containerID="de875bc2b4b48038cb312559cbeb5112ddb1b13cd8a1571d5043bbfea1a5eb40" Oct 30 22:23:36.433423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.433405 4292 scope.go:122] "RemoveContainer" containerID="7349dda278dcd87008698b0cf68afcc70d9ded8e0cea5d1d80cb91344cfc4755" Oct 30 22:23:36.465061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.465038 4292 scope.go:122] "RemoveContainer" containerID="948016b98fce10e2c086afaab9ff71eb0b3201a03e282130490f5c9e8e8d2c05" Oct 30 22:23:36.481325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.481305 4292 scope.go:122] "RemoveContainer" containerID="5002f8008d1da5b909fe078684fe12a79aa3092ed55aae57ecc94c8b555e8e27" Oct 30 22:23:36.490971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.490950 4292 scope.go:122] "RemoveContainer" containerID="ae4d2a46f3565f92bb4970b414b4fae023e7a7bbe6309896e3b7b8b145e80ee6" Oct 30 22:23:36.499766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.499748 4292 scope.go:122] "RemoveContainer" containerID="311729011fee6e667676d7e883f3ceebfd837889ea04776cbe7fc58c017697fb" Oct 30 22:23:36.508692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.508677 4292 scope.go:122] "RemoveContainer" containerID="52377c8232c2b732fa9e787a13d66967edd3b4b50f6cf554e2d70708ea2e183e" Oct 30 22:23:36.517389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.517372 4292 scope.go:122] "RemoveContainer" containerID="5ec2ee5a35e91c51196f7ee14c6173505a22a74b45e7b8d07ebdce840402d57e" Oct 30 22:23:36.530641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:36.529282 4292 scope.go:122] "RemoveContainer" containerID="30de657658caaf4268527fc58c626a81c92f073d1dcf5a8339529523ea956115" Oct 30 22:23:37.275267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.275236 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:23:37.275455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.275416 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" containerID="cri-o://86a81dd70017384861bbd2411ffe8341f4d4ec9ae2275df3e2a843d4e7793024" gracePeriod=30 Oct 30 22:23:37.316195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.316160 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:23:37.316436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.316397 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" containerID="cri-o://e25d830a4f8951b4d2608fec8737378b6b2b72c0b93b157abbf34ff4d3749b89" gracePeriod=30 Oct 30 22:23:37.714405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.714365 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:23:37.714913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:37.714616 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" containerID="cri-o://1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f" gracePeriod=30 Oct 30 22:23:38.819843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:38.819799 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk"] Oct 30 22:23:38.860684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:38.860650 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-05eb684c-h9rnk"] Oct 30 22:23:38.867670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:38.867643 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:23:38.867972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:38.867928 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" containerID="cri-o://5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57" gracePeriod=30 Oct 30 22:23:40.152466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:40.152430 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd85bc7-bfa7-4bba-8659-d553ba579cb5" path="/var/lib/kubelet/pods/bcd85bc7-bfa7-4bba-8659-d553ba579cb5/volumes" Oct 30 22:23:40.494323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:40.494218 4292 patch_prober.go:29] interesting pod/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:40.494323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:40.494297 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:41.014251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:41.013678 4292 patch_prober.go:29] interesting pod/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:41.014251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:41.013895 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:42.480434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.480399 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:42.480809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.480466 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:42.525642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.525402 4292 generic.go:356] "Generic (PLEG): container finished" podID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerID="e25d830a4f8951b4d2608fec8737378b6b2b72c0b93b157abbf34ff4d3749b89" exitCode=0 Oct 30 22:23:42.525642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.525480 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" event={"ID":"5d90424d-40d1-4bfd-9780-b3b36b0be4ff","Type":"ContainerDied","Data":"e25d830a4f8951b4d2608fec8737378b6b2b72c0b93b157abbf34ff4d3749b89"} Oct 30 22:23:42.527877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.527756 4292 generic.go:356] "Generic (PLEG): container finished" podID="b96bd55b-3272-406e-bb54-11d797566ef4" containerID="86a81dd70017384861bbd2411ffe8341f4d4ec9ae2275df3e2a843d4e7793024" exitCode=0 Oct 30 22:23:42.527877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.527790 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" event={"ID":"b96bd55b-3272-406e-bb54-11d797566ef4","Type":"ContainerDied","Data":"86a81dd70017384861bbd2411ffe8341f4d4ec9ae2275df3e2a843d4e7793024"} Oct 30 22:23:42.622134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.622099 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:23:42.651523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.651501 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:23:42.660006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.659970 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660031 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660071 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660111 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660165 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660246 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.660400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660335 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660359 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.660400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660364 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660451 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660495 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660516 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660546 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660593 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:42.660634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660615 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660693 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660743 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.660867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660788 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\") pod \"b96bd55b-3272-406e-bb54-11d797566ef4\" (UID: \"b96bd55b-3272-406e-bb54-11d797566ef4\") " Oct 30 22:23:42.660867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660827 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.661078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.660870 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.661078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661009 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\") pod \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\" (UID: \"5d90424d-40d1-4bfd-9780-b3b36b0be4ff\") " Oct 30 22:23:42.661191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661129 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.661449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661425 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:42.662032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661824 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.662032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661916 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.662032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.661990 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.662032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662012 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.662303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662242 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662264 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96bd55b-3272-406e-bb54-11d797566ef4-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662282 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662297 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662311 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662324 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662334 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662347 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662363 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.662543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.662376 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.663214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.663189 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "kube-api-access-42cpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.663427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.663403 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:42.663954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.663928 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.663954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.663941 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token" pod "b96bd55b-3272-406e-bb54-11d797566ef4" (UID: "b96bd55b-3272-406e-bb54-11d797566ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.664117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.663989 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:42.664117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.664001 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "kube-api-access-58sf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.664303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.664285 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.665216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.665198 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy" pod "5d90424d-40d1-4bfd-9780-b3b36b0be4ff" (UID: "5d90424d-40d1-4bfd-9780-b3b36b0be4ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.764127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764095 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764123 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764137 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96bd55b-3272-406e-bb54-11d797566ef4-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764151 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cpq\" (UniqueName: \"kubernetes.io/projected/b96bd55b-3272-406e-bb54-11d797566ef4-kube-api-access-42cpq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764164 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764177 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764190 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sf6\" (UniqueName: \"kubernetes.io/projected/5d90424d-40d1-4bfd-9780-b3b36b0be4ff-kube-api-access-58sf6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.764285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.764203 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96bd55b-3272-406e-bb54-11d797566ef4-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.902273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.902238 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903311 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" containerName="payload-pre-processing" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903714 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" containerName="payload-pre-processing" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903737 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2b6e8785-0a0c-4072-8033-4690e2148780" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903745 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6e8785-0a0c-4072-8033-4690e2148780" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903766 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903774 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903813 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903820 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903834 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903841 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903880 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" containerName="payload-processing" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.903887 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" containerName="payload-processing" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904117 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67caa40-2a33-44a8-80e0-ae9f8ecb2cc2" containerName="payload-pre-processing" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904133 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6e8785-0a0c-4072-8033-4690e2148780" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904152 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904166 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" containerName="revoke-keys" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904175 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" containerName="istio-proxy" Oct 30 22:23:42.904698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.904190 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a80a521-b11f-4fe0-a73c-c7aaffed564d" containerName="payload-processing" Oct 30 22:23:42.916463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.914743 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.927986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.927921 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:23:42.939225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.938788 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970189 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gls8m\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970562 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-367f4491-openshift-default-dockercfg-d7tpf\"" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970722 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970756 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970783 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970854 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970883 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970911 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnd8w\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.971152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.970966 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.971769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.971626 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.971769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.971692 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.971769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.971758 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.971921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.971858 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.971921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.971906 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.972213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.972173 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.972213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.972210 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.972384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.972268 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.972384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.972321 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:42.972483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.972390 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:42.993332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.993310 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:23:42.998520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:42.998495 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:23:43.040860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.040831 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:23:43.073597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.073542 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.073765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.073711 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.074023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.073994 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:43.074152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.074132 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.074644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.074620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.075035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075009 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.075413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.075533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075471 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.075533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075499 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.075746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075595 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\") pod \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\" (UID: \"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e\") " Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075902 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.075946 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076075 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076177 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076209 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnd8w\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076269 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076334 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076384 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076407 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076471 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076604 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076656 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076740 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076768 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076796 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076834 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076893 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.076967 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.077010 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gls8m\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.077088 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.077161 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.077240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.077183 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.078553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.078000 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "kube-api-access-72hp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:43.078553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.078157 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:43.078553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.078356 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.078553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.078388 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.078860 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.079085 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.079606 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.079807 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.079831 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.080050 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.080061 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.080321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.080282 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.080794 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.080389 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.082139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.082110 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.082276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.082256 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:43.082471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.082451 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.084064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.083977 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.084561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.084539 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.092660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.092026 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token" pod "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" (UID: "f1d0c3d0-48b2-44fc-a24f-f9df4da3583e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:43.142587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.142503 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.143084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.142980 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.143264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.143114 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnd8w\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.145735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.145711 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gls8m\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.178362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178330 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178360 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178370 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178381 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178390 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hp8\" (UniqueName: \"kubernetes.io/projected/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-kube-api-access-72hp8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178399 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.178565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.178408 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.245168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.245127 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:43.261896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.261833 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:43.538933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.538901 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" Oct 30 22:23:43.539352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.538950 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5" event={"ID":"5d90424d-40d1-4bfd-9780-b3b36b0be4ff","Type":"ContainerDied","Data":"01fe347a42b0d6ec21e52aa39eb4e1ffbf19bc6e1e00d2557c6b9bc4e3504676"} Oct 30 22:23:43.539352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.539014 4292 scope.go:122] "RemoveContainer" containerID="e25d830a4f8951b4d2608fec8737378b6b2b72c0b93b157abbf34ff4d3749b89" Oct 30 22:23:43.540928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.540902 4292 generic.go:356] "Generic (PLEG): container finished" podID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerID="1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f" exitCode=0 Oct 30 22:23:43.541029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.540953 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" Oct 30 22:23:43.541029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.540980 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" event={"ID":"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e","Type":"ContainerDied","Data":"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f"} Oct 30 22:23:43.541118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.541029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm" event={"ID":"f1d0c3d0-48b2-44fc-a24f-f9df4da3583e","Type":"ContainerDied","Data":"2b50d7e04fea4471158d533868b35c88275b6be23ec7ba66979ab879dcb60af4"} Oct 30 22:23:43.542496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.542474 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" event={"ID":"b96bd55b-3272-406e-bb54-11d797566ef4","Type":"ContainerDied","Data":"ce2083fbd90ca04a7b8c3b2e6980ad1a2348e4e821ceca1d5d5e0e646a11ca80"} Oct 30 22:23:43.542635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.542528 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g" Oct 30 22:23:43.552356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.552339 4292 scope.go:122] "RemoveContainer" containerID="1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f" Oct 30 22:23:43.563415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.563398 4292 scope.go:122] "RemoveContainer" containerID="1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f" Oct 30 22:23:43.563722 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:43.563702 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f\": container with ID starting with 1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f not found: ID does not exist" containerID="1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f" Oct 30 22:23:43.563803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.563730 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f"} err="failed to get container status \"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f\": rpc error: code = NotFound desc = could not find container \"1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f\": container with ID starting with 1280d4a73a016ab0b51c5bd3eb921339986eeed59c7f620b2979dc3d6fe9ff6f not found: ID does not exist" Oct 30 22:23:43.563803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.563749 4292 scope.go:122] "RemoveContainer" containerID="86a81dd70017384861bbd2411ffe8341f4d4ec9ae2275df3e2a843d4e7793024" Oct 30 22:23:43.711904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.711814 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:23:43.720222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.720195 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-vcj5g"] Oct 30 22:23:43.725682 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:43.725642 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37ea491_7249_4452_93c8_d9bd30cddc76.slice/crio-3f988acc5d53f108f4306525383460f2b42d2853094a4a2a29a9563322a65bee WatchSource:0}: Error finding container 3f988acc5d53f108f4306525383460f2b42d2853094a4a2a29a9563322a65bee: Status 404 returned error can't find the container with id 3f988acc5d53f108f4306525383460f2b42d2853094a4a2a29a9563322a65bee Oct 30 22:23:43.727959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.727930 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.727959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.727939 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.728107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.728002 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.728107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.728010 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.728107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.728046 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.728107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.728052 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:23:43.762122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.762096 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:23:43.766084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.766060 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:23:43.831417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.831394 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:23:43.843479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.843453 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gcvh5"] Oct 30 22:23:43.959931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.959899 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:23:43.967352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:43.965421 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-wwmvm"] Oct 30 22:23:44.161986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.161950 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d90424d-40d1-4bfd-9780-b3b36b0be4ff" path="/var/lib/kubelet/pods/5d90424d-40d1-4bfd-9780-b3b36b0be4ff/volumes" Oct 30 22:23:44.163143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.163102 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96bd55b-3272-406e-bb54-11d797566ef4" path="/var/lib/kubelet/pods/b96bd55b-3272-406e-bb54-11d797566ef4/volumes" Oct 30 22:23:44.164066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.164045 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" path="/var/lib/kubelet/pods/f1d0c3d0-48b2-44fc-a24f-f9df4da3583e/volumes" Oct 30 22:23:44.220064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.219983 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:23:44.311658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.311627 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.311866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.311848 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312025 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312164 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:44.312337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312315 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312416 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312564 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312641 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312795 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.312909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312880 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:44.312988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.312923 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\") pod \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\" (UID: \"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4\") " Oct 30 22:23:44.313283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.313259 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:44.313742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.313456 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:44.314401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.314351 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.314401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.314380 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.314531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.314404 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.314531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.314379 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:44.314531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.314421 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.316552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.316525 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:44.316673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.316558 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:44.316673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.316607 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "kube-api-access-fgxrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:44.316809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.316742 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo" pod "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" (UID: "cb7c79ed-c3a6-45d1-ae79-579ad961ecb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:44.416916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.415926 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.416916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.415968 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxrx\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-kube-api-access-fgxrx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.416916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.415986 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.416916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.416000 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.416916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.416013 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.555389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.555298 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" event={"ID":"28bef028-011b-4770-a463-b56c93bdc705","Type":"ContainerStarted","Data":"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3"} Oct 30 22:23:44.555814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.555396 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" event={"ID":"28bef028-011b-4770-a463-b56c93bdc705","Type":"ContainerStarted","Data":"e36656c7de2ad26414ef0842ddc85d8d3f3119ca7269169a00c81106919648a3"} Oct 30 22:23:44.557250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.557217 4292 generic.go:356] "Generic (PLEG): container finished" podID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerID="5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57" exitCode=0 Oct 30 22:23:44.557414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.557327 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" Oct 30 22:23:44.557522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.557415 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerDied","Data":"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57"} Oct 30 22:23:44.557522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.557472 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj" event={"ID":"cb7c79ed-c3a6-45d1-ae79-579ad961ecb4","Type":"ContainerDied","Data":"0969503f0b6d163c095e2e6494fda6ba0ad7bd8d0e29d45eb10832c1d115f59d"} Oct 30 22:23:44.557522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.557504 4292 scope.go:122] "RemoveContainer" containerID="5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57" Oct 30 22:23:44.564569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.564549 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" event={"ID":"c37ea491-7249-4452-93c8-d9bd30cddc76","Type":"ContainerStarted","Data":"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612"} Oct 30 22:23:44.564716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.564702 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" event={"ID":"c37ea491-7249-4452-93c8-d9bd30cddc76","Type":"ContainerStarted","Data":"3f988acc5d53f108f4306525383460f2b42d2853094a4a2a29a9563322a65bee"} Oct 30 22:23:44.573328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.573310 4292 scope.go:122] "RemoveContainer" containerID="75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777" Oct 30 22:23:44.582994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.582927 4292 scope.go:122] "RemoveContainer" containerID="5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57" Oct 30 22:23:44.583223 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:44.583201 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57\": container with ID starting with 5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57 not found: ID does not exist" containerID="5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57" Oct 30 22:23:44.583334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.583262 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57"} err="failed to get container status \"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57\": rpc error: code = NotFound desc = could not find container \"5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57\": container with ID starting with 5d20489c3e3eb83f64f5042785641fd496f3c4c761251b3140264ff238e37c57 not found: ID does not exist" Oct 30 22:23:44.583334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.583285 4292 scope.go:122] "RemoveContainer" containerID="75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777" Oct 30 22:23:44.583611 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:44.583559 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777\": container with ID starting with 75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777 not found: ID does not exist" containerID="75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777" Oct 30 22:23:44.583679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.583618 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777"} err="failed to get container status \"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777\": rpc error: code = NotFound desc = could not find container \"75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777\": container with ID starting with 75f0b451cc4627a5add3e4873061bf8640f9968eec730646d74c735d94fc2777 not found: ID does not exist" Oct 30 22:23:44.718598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.717554 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podStartSLOduration=2.71753096 podStartE2EDuration="2.71753096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:44.693141064 +0000 UTC m=+1752.661960629" watchObservedRunningTime="2026-10-30 22:23:44.71753096 +0000 UTC m=+1752.686350524" Oct 30 22:23:44.824655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.824491 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podStartSLOduration=2.824468985 podStartE2EDuration="2.824468985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:44.821080653 +0000 UTC m=+1752.789900219" watchObservedRunningTime="2026-10-30 22:23:44.824468985 +0000 UTC m=+1752.793288551" Oct 30 22:23:44.943936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.943881 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:23:44.948391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:44.948362 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-05eb684c-gw-openshift-default-795cdfbd57-9kmjj"] Oct 30 22:23:45.245682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.245562 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:45.247537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.247509 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:45.247666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.247562 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:45.262718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.262695 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:45.263878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.263855 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:45.263967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:45.263909 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:46.153251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:46.153211 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" path="/var/lib/kubelet/pods/cb7c79ed-c3a6-45d1-ae79-579ad961ecb4/volumes" Oct 30 22:23:46.246417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:46.246378 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:46.246694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:46.246654 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:46.262097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:46.262068 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:46.262216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:46.262116 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:47.245943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:47.245901 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:47.246331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:47.245960 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:47.262371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:47.262345 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:47.262502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:47.262389 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:48.245715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:48.245676 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:48.245878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:48.245750 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:48.262078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:48.262054 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:48.262389 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:48.262110 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:49.246077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:49.246040 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:49.246267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:49.246100 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:49.263007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:49.262974 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:49.263372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:49.263024 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:50.000282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.000246 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:23:50.001692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.001664 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.001868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.001853 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.002032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.002021 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" Oct 30 22:23:50.002109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.002098 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" Oct 30 22:23:50.002387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.002369 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.002503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.002493 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d0c3d0-48b2-44fc-a24f-f9df4da3583e" containerName="istio-proxy" Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.256339 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.256397 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.259909 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.262989 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.263046 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 22:23:50.273622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.263171 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.291030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.288413 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr9r\" (UniqueName: \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.291030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.288713 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.312265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.312220 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:23:50.313333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.313303 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.313333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.313331 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.313632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.313613 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7c79ed-c3a6-45d1-ae79-579ad961ecb4" containerName="istio-proxy" Oct 30 22:23:50.318901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.318862 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:23:50.319320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.319296 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8e1227\"" Oct 30 22:23:50.320431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.319995 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.367824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.367793 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8e1227-dockercfg-qdk9q\"" Oct 30 22:23:50.391401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.390898 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr9r\" (UniqueName: \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.391401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.391120 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjkc\" (UniqueName: \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.391401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.391172 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.393499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.391892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.393499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.392945 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.457593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.457538 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr9r\" (UniqueName: \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\") pod \"payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.495419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.493638 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjkc\" (UniqueName: \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.495419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.493700 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.495419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.494870 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.558941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.558904 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjkc\" (UniqueName: \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\") pod \"payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:50.606444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.606407 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:50.639029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:50.638154 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:51.032054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.032019 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:23:51.063071 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:51.063036 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10882900_cceb_48dd_bf40_486a132bed75.slice/crio-e6431651144bac49ee41f3735804f8481e0bd9db8f607c5b5b847d667bd305f4 WatchSource:0}: Error finding container e6431651144bac49ee41f3735804f8481e0bd9db8f607c5b5b847d667bd305f4: Status 404 returned error can't find the container with id e6431651144bac49ee41f3735804f8481e0bd9db8f607c5b5b847d667bd305f4 Oct 30 22:23:51.078663 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:23:51.078636 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6441d57_ca7e_4f63_9dc5_6e0088bd81c0.slice/crio-29939cdbe5e38b9b1d01ffd07446a3717a1bfc3e6ea0085ef862a773eab78e37 WatchSource:0}: Error finding container 29939cdbe5e38b9b1d01ffd07446a3717a1bfc3e6ea0085ef862a773eab78e37: Status 404 returned error can't find the container with id 29939cdbe5e38b9b1d01ffd07446a3717a1bfc3e6ea0085ef862a773eab78e37 Oct 30 22:23:51.079033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.079013 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:23:51.251262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.251236 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:51.251699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.251669 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:51.252388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.252373 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:23:51.267338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.267309 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:51.267556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.267533 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:51.268235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.268221 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:23:51.798918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.798877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" event={"ID":"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0","Type":"ContainerStarted","Data":"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231"} Oct 30 22:23:51.799438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.798982 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" event={"ID":"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0","Type":"ContainerStarted","Data":"29939cdbe5e38b9b1d01ffd07446a3717a1bfc3e6ea0085ef862a773eab78e37"} Oct 30 22:23:51.799438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.799152 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:23:51.800982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.800955 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" event={"ID":"10882900-cceb-48dd-bf40-486a132bed75","Type":"ContainerStarted","Data":"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00"} Oct 30 22:23:51.801113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.801008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" event={"ID":"10882900-cceb-48dd-bf40-486a132bed75","Type":"ContainerStarted","Data":"e6431651144bac49ee41f3735804f8481e0bd9db8f607c5b5b847d667bd305f4"} Oct 30 22:23:51.801113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.801049 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:23:51.926647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:51.926588 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" podStartSLOduration=1.926553646 podStartE2EDuration="1.926553646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:51.922123302 +0000 UTC m=+1759.890942865" watchObservedRunningTime="2026-10-30 22:23:51.926553646 +0000 UTC m=+1759.895373214" Oct 30 22:23:52.045030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:52.044931 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" podStartSLOduration=3.044912277 podStartE2EDuration="3.044912277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:52.044560049 +0000 UTC m=+1760.013379615" watchObservedRunningTime="2026-10-30 22:23:52.044912277 +0000 UTC m=+1760.013731842" Oct 30 22:23:52.584236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:52.584200 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:23:52.584421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:52.584403 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" containerID="cri-o://0a30f6576a42f934140736cf217af57b602322bba70c7add156b0c20f4092e58" gracePeriod=30 Oct 30 22:23:52.860405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:52.860329 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:23:52.860860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:52.860496 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" containerID="cri-o://7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68" gracePeriod=30 Oct 30 22:23:53.541650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:53.541613 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:23:53.541822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:53.541796 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" containerID="cri-o://3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b" gracePeriod=30 Oct 30 22:23:56.641413 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:56.641375 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:23:57.428027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.427986 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:57.428207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.428062 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:57.863350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.863312 4292 generic.go:356] "Generic (PLEG): container finished" podID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerID="0a30f6576a42f934140736cf217af57b602322bba70c7add156b0c20f4092e58" exitCode=0 Oct 30 22:23:57.863826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.863370 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" event={"ID":"68d1959d-c544-483c-a7fc-0e0ec67f3629","Type":"ContainerDied","Data":"0a30f6576a42f934140736cf217af57b602322bba70c7add156b0c20f4092e58"} Oct 30 22:23:57.863826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.863423 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" event={"ID":"68d1959d-c544-483c-a7fc-0e0ec67f3629","Type":"ContainerDied","Data":"fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce"} Oct 30 22:23:57.863826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.863453 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9b1c9b57930a4a5222e4bad35f225dc8593c7ebf59e8222512f6a609a906ce" Oct 30 22:23:57.878015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:57.877832 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:23:58.052335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052298 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.052502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052366 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.052502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052439 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052746 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052920 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052968 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.052995 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.053065 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.054739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.053097 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\") pod \"68d1959d-c544-483c-a7fc-0e0ec67f3629\" (UID: \"68d1959d-c544-483c-a7fc-0e0ec67f3629\") " Oct 30 22:23:58.055180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.055094 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.055453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.055429 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:58.055726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.055696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.055967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.055885 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.056147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.056064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.065780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.063437 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "kube-api-access-l46rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:58.076645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.071775 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:58.076645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.072957 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:58.076645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.076614 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy" pod "68d1959d-c544-483c-a7fc-0e0ec67f3629" (UID: "68d1959d-c544-483c-a7fc-0e0ec67f3629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.138479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.138454 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:23:58.155822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155794 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155825 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155842 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155856 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46rs\" (UniqueName: \"kubernetes.io/projected/68d1959d-c544-483c-a7fc-0e0ec67f3629-kube-api-access-l46rs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155870 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155884 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1959d-c544-483c-a7fc-0e0ec67f3629-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155898 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155912 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.155949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.155927 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1959d-c544-483c-a7fc-0e0ec67f3629-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.257251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257159 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257213 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257297 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257455 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.257600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257530 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257544 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.257677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257611 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257637 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257678 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257756 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.257874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257787 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.257874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.257798 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\") pod \"dce09c22-dcd7-4141-a38c-a747590c3e89\" (UID: \"dce09c22-dcd7-4141-a38c-a747590c3e89\") " Oct 30 22:23:58.258052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.258024 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.258174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.258154 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:58.259275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259248 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.259388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259278 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce09c22-dcd7-4141-a38c-a747590c3e89-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.259388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259294 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.259388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259309 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.259388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259323 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.259905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.259884 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "kube-api-access-v97mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:58.260057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.260034 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:58.260125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.260053 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.260326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.260311 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token" pod "dce09c22-dcd7-4141-a38c-a747590c3e89" (UID: "dce09c22-dcd7-4141-a38c-a747590c3e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:58.362362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.362295 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97mr\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-kube-api-access-v97mr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.362362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.362358 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.362623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.362375 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.362623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.362406 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce09c22-dcd7-4141-a38c-a747590c3e89-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.490644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.490608 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:58.490819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.490679 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:58.819876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.819817 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:23:58.882569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.882532 4292 generic.go:356] "Generic (PLEG): container finished" podID="c9721985-ff96-4206-ab76-70cad8b4d602" containerID="3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b" exitCode=0 Oct 30 22:23:58.883016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.882674 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" event={"ID":"c9721985-ff96-4206-ab76-70cad8b4d602","Type":"ContainerDied","Data":"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b"} Oct 30 22:23:58.883016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.882774 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" event={"ID":"c9721985-ff96-4206-ab76-70cad8b4d602","Type":"ContainerDied","Data":"c376a42d628c00c1cc1df123dff7b183a0826dc81a097d6d0aac93d96f41a60f"} Oct 30 22:23:58.883016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.882685 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5" Oct 30 22:23:58.883016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.882824 4292 scope.go:122] "RemoveContainer" containerID="3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b" Oct 30 22:23:58.884896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.884870 4292 generic.go:356] "Generic (PLEG): container finished" podID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerID="7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68" exitCode=0 Oct 30 22:23:58.885023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.884928 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" Oct 30 22:23:58.885023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.884998 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" event={"ID":"dce09c22-dcd7-4141-a38c-a747590c3e89","Type":"ContainerDied","Data":"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68"} Oct 30 22:23:58.885129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.885042 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m" event={"ID":"dce09c22-dcd7-4141-a38c-a747590c3e89","Type":"ContainerDied","Data":"412f13acffbed2ec169b33dd6505147e97af947fd011a6040a0df0751b3d8e06"} Oct 30 22:23:58.885454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.885439 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t" Oct 30 22:23:58.896200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.896183 4292 scope.go:122] "RemoveContainer" containerID="3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b" Oct 30 22:23:58.896481 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:58.896457 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b\": container with ID starting with 3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b not found: ID does not exist" containerID="3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b" Oct 30 22:23:58.896547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.896488 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b"} err="failed to get container status \"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b\": rpc error: code = NotFound desc = could not find container \"3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b\": container with ID starting with 3d78e39d23e65858cb12d54b0e6420aca68417ceeb36baf08d347ba69bf9ad2b not found: ID does not exist" Oct 30 22:23:58.896547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.896505 4292 scope.go:122] "RemoveContainer" containerID="7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68" Oct 30 22:23:58.906012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.905994 4292 scope.go:122] "RemoveContainer" containerID="7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68" Oct 30 22:23:58.906281 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:23:58.906263 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68\": container with ID starting with 7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68 not found: ID does not exist" containerID="7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68" Oct 30 22:23:58.906321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.906288 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68"} err="failed to get container status \"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68\": rpc error: code = NotFound desc = could not find container \"7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68\": container with ID starting with 7585645ab825b7e7000ed9514e27b4faa13909ac4b8b88dec56b8bdaa0635f68 not found: ID does not exist" Oct 30 22:23:58.970378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970356 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970508 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970645 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970709 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:58.970824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970739 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970829 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.970881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970858 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.971013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970886 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.971013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970919 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.971013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.970994 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.971179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971028 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\") pod \"c9721985-ff96-4206-ab76-70cad8b4d602\" (UID: \"c9721985-ff96-4206-ab76-70cad8b4d602\") " Oct 30 22:23:58.971179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971123 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.971373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971346 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.971837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971812 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.971927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971843 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.971927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971860 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721985-ff96-4206-ab76-70cad8b4d602-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.971927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971876 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.971927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.971891 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:58.973089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.973064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:58.973182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.973103 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:58.973182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.973152 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:58.973258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:58.973187 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp" pod "c9721985-ff96-4206-ab76-70cad8b4d602" (UID: "c9721985-ff96-4206-ab76-70cad8b4d602"). InnerVolumeSpecName "kube-api-access-zmznp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:59.073167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.073086 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721985-ff96-4206-ab76-70cad8b4d602-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:59.073167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.073113 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:59.073167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.073124 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmznp\" (UniqueName: \"kubernetes.io/projected/c9721985-ff96-4206-ab76-70cad8b4d602-kube-api-access-zmznp\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:59.073167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.073134 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721985-ff96-4206-ab76-70cad8b4d602-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:59.190560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.190530 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:23:59.197785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.197762 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-vv52m"] Oct 30 22:23:59.310739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.310709 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:23:59.319299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.319269 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-wn29t"] Oct 30 22:23:59.433417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.433300 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:23:59.438413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:23:59.438389 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-f2kv5"] Oct 30 22:24:00.148282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:00.148250 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" path="/var/lib/kubelet/pods/68d1959d-c544-483c-a7fc-0e0ec67f3629/volumes" Oct 30 22:24:00.148828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:00.148812 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" path="/var/lib/kubelet/pods/c9721985-ff96-4206-ab76-70cad8b4d602/volumes" Oct 30 22:24:00.149274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:00.149261 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" path="/var/lib/kubelet/pods/dce09c22-dcd7-4141-a38c-a747590c3e89/volumes" Oct 30 22:24:02.811748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:02.811721 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:24:02.812126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:02.812105 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:24:03.582832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:03.582798 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn"] Oct 30 22:24:03.586709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:03.586689 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-678b1628-jhzkn"] Oct 30 22:24:03.676672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:03.676636 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:24:03.677016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:03.676987 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" containerID="cri-o://8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83" gracePeriod=30 Oct 30 22:24:04.148811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:04.148779 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6e8785-0a0c-4072-8033-4690e2148780" path="/var/lib/kubelet/pods/2b6e8785-0a0c-4072-8033-4690e2148780/volumes" Oct 30 22:24:07.718880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.718836 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:24:07.719943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.719917 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" Oct 30 22:24:07.719943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.719944 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" Oct 30 22:24:07.720140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720026 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" Oct 30 22:24:07.720140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720033 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" Oct 30 22:24:07.720140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720053 4292 cpu_manager.go:414] "containerMap: removing container" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" Oct 30 22:24:07.720140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720060 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" Oct 30 22:24:07.720342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720212 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce09c22-dcd7-4141-a38c-a747590c3e89" containerName="istio-proxy" Oct 30 22:24:07.720342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720223 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d1959d-c544-483c-a7fc-0e0ec67f3629" containerName="istio-proxy" Oct 30 22:24:07.720342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.720237 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9721985-ff96-4206-ab76-70cad8b4d602" containerName="istio-proxy" Oct 30 22:24:07.726121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.726097 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:07.791337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.791299 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:24:07.842851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.842820 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-6c7d3f-dockercfg-9s999\"" Oct 30 22:24:07.843175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.843156 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-6c7d3f\"" Oct 30 22:24:07.875251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.875214 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6hs\" (UniqueName: \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:07.876011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.875979 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:07.911287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.911244 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:24:07.925967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.925935 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:07.938003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.937971 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:24:07.978719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.977459 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6hs\" (UniqueName: \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:07.978719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.977674 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:07.978719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.977805 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2d2\" (UniqueName: \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:07.978719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.977839 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:07.979087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:07.979068 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:08.049623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.043882 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6hs\" (UniqueName: \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\") pod \"payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:08.080675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.080633 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2d2\" (UniqueName: \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:08.080887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.080692 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:08.081940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.081914 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:08.145808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.145776 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2d2\" (UniqueName: \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\") pod \"payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:08.257335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.257240 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:08.344378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.343079 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:08.644369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.644341 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:24:08.645910 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:08.645875 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89aa5a36_166d_4e7c_a318_ae75b9d90085.slice/crio-213a56ded25f57bb29f9b52cf193b8ca0efd4060546db4354abeac7e1ec891dc WatchSource:0}: Error finding container 213a56ded25f57bb29f9b52cf193b8ca0efd4060546db4354abeac7e1ec891dc: Status 404 returned error can't find the container with id 213a56ded25f57bb29f9b52cf193b8ca0efd4060546db4354abeac7e1ec891dc Oct 30 22:24:08.773102 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:08.773077 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a804b4b_4475_47fa_9181_acf2327826e3.slice/crio-fe04fd128c2d70b43c38931eeef69eb2d9133da69918ecbf81ca66813272cf3c WatchSource:0}: Error finding container fe04fd128c2d70b43c38931eeef69eb2d9133da69918ecbf81ca66813272cf3c: Status 404 returned error can't find the container with id fe04fd128c2d70b43c38931eeef69eb2d9133da69918ecbf81ca66813272cf3c Oct 30 22:24:08.773394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.773214 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:24:08.957804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.957781 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:24:08.996422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996391 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996442 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996524 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996643 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:08.996684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996676 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996742 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996790 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996908 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.996999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.996953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:08.997106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.997091 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.997212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.997199 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\") pod \"b1df68f5-165d-4876-9379-27b2e703e508\" (UID: \"b1df68f5-165d-4876-9379-27b2e703e508\") " Oct 30 22:24:08.997422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.997192 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:08.997422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.997363 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:08.997585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.997550 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:08.998042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.998020 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.998099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.998051 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.998099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.998068 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1df68f5-165d-4876-9379-27b2e703e508-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.998099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.998082 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.998099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.998096 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.999166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.999144 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:08.999275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.999219 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:08.999275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.999256 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:08.999449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:08.999431 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx" pod "b1df68f5-165d-4876-9379-27b2e703e508" (UID: "b1df68f5-165d-4876-9379-27b2e703e508"). InnerVolumeSpecName "kube-api-access-2nxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:09.081514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.081480 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" event={"ID":"89aa5a36-166d-4e7c-a318-ae75b9d90085","Type":"ContainerStarted","Data":"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4"} Oct 30 22:24:09.081627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.081546 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:09.081627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.081601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" event={"ID":"89aa5a36-166d-4e7c-a318-ae75b9d90085","Type":"ContainerStarted","Data":"213a56ded25f57bb29f9b52cf193b8ca0efd4060546db4354abeac7e1ec891dc"} Oct 30 22:24:09.084469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.084442 4292 generic.go:356] "Generic (PLEG): container finished" podID="b1df68f5-165d-4876-9379-27b2e703e508" containerID="8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83" exitCode=0 Oct 30 22:24:09.084599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.084505 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" Oct 30 22:24:09.084599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.084521 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerDied","Data":"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83"} Oct 30 22:24:09.084729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.084672 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n" event={"ID":"b1df68f5-165d-4876-9379-27b2e703e508","Type":"ContainerDied","Data":"26a02069353683fc5b811bf11b3fc3a5d694de767f2be0769f457d3803e04052"} Oct 30 22:24:09.085097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.085075 4292 scope.go:122] "RemoveContainer" containerID="8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83" Oct 30 22:24:09.087565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.086905 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" event={"ID":"3a804b4b-4475-47fa-9181-acf2327826e3","Type":"ContainerStarted","Data":"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623"} Oct 30 22:24:09.087565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.086959 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" event={"ID":"3a804b4b-4475-47fa-9181-acf2327826e3","Type":"ContainerStarted","Data":"fe04fd128c2d70b43c38931eeef69eb2d9133da69918ecbf81ca66813272cf3c"} Oct 30 22:24:09.087565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.087077 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:09.098818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.098718 4292 scope.go:122] "RemoveContainer" containerID="fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6" Oct 30 22:24:09.099333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.099311 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1df68f5-165d-4876-9379-27b2e703e508-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:09.099431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.099337 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:09.099431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.099353 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1df68f5-165d-4876-9379-27b2e703e508-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:09.099431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.099369 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxmx\" (UniqueName: \"kubernetes.io/projected/b1df68f5-165d-4876-9379-27b2e703e508-kube-api-access-2nxmx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:09.111215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.111195 4292 scope.go:122] "RemoveContainer" containerID="8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83" Oct 30 22:24:09.111547 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:09.111525 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83\": container with ID starting with 8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83 not found: ID does not exist" containerID="8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83" Oct 30 22:24:09.111701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.111552 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83"} err="failed to get container status \"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83\": rpc error: code = NotFound desc = could not find container \"8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83\": container with ID starting with 8813d61576560b76ca41f9377d3cdb6dd4f73ddad430284f1222cf70282f6f83 not found: ID does not exist" Oct 30 22:24:09.111701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.111648 4292 scope.go:122] "RemoveContainer" containerID="fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6" Oct 30 22:24:09.111945 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:09.111918 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6\": container with ID starting with fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6 not found: ID does not exist" containerID="fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6" Oct 30 22:24:09.112017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.111953 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6"} err="failed to get container status \"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6\": rpc error: code = NotFound desc = could not find container \"fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6\": container with ID starting with fd32fe3776a7b8fb5af1f29b211419a6bcbb89f4cbeb089abb93613f2292ceb6 not found: ID does not exist" Oct 30 22:24:09.203893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.203802 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" podStartSLOduration=2.203789656 podStartE2EDuration="2.203789656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:09.20260557 +0000 UTC m=+1777.171425154" watchObservedRunningTime="2026-10-30 22:24:09.203789656 +0000 UTC m=+1777.172609220" Oct 30 22:24:09.323392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.323333 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:24:09.338387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.333662 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-678b1628-openshift-default-5bff7fc8f7-gw57n"] Oct 30 22:24:09.448222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:09.448158 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" podStartSLOduration=2.448141451 podStartE2EDuration="2.448141451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:09.443937144 +0000 UTC m=+1777.412756727" watchObservedRunningTime="2026-10-30 22:24:09.448141451 +0000 UTC m=+1777.416961015" Oct 30 22:24:10.148356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:10.148316 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1df68f5-165d-4876-9379-27b2e703e508" path="/var/lib/kubelet/pods/b1df68f5-165d-4876-9379-27b2e703e508/volumes" Oct 30 22:24:13.203973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.203937 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:24:13.204654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.204615 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:13.204654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.204631 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:13.204654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.204648 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:13.204654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.204653 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:13.204815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.204799 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:13.210023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.210003 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258124 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258181 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258277 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258605 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258672 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258815 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258909 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.258988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.258975 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rrr\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.340489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.340455 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:24:13.360238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.360423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360269 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.360423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360346 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rrr\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.360636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360612 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.360849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360827 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.360984 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361081 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361220 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361261 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361358 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361770 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.361876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.361809 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.362073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.362049 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.362210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.362182 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.363587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.363556 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.364009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.363987 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.424864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.424832 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rrr\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.424864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.424849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\") pod \"e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.524804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.524737 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:13.996481 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:13.996448 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905153d1_5e1a_4828_9974_84c54b150d64.slice/crio-50c4623e49a1dc0b9a3f852de543dc9262d245a29d3da4595b11b6391d71041c WatchSource:0}: Error finding container 50c4623e49a1dc0b9a3f852de543dc9262d245a29d3da4595b11b6391d71041c: Status 404 returned error can't find the container with id 50c4623e49a1dc0b9a3f852de543dc9262d245a29d3da4595b11b6391d71041c Oct 30 22:24:13.996918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.996882 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:24:13.999115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.999081 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:13.999208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.999152 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:13.999208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:13.999185 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:14.132773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:14.132724 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" event={"ID":"905153d1-5e1a-4828-9974-84c54b150d64","Type":"ContainerStarted","Data":"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b"} Oct 30 22:24:14.132940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:14.132808 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" event={"ID":"905153d1-5e1a-4828-9974-84c54b150d64","Type":"ContainerStarted","Data":"50c4623e49a1dc0b9a3f852de543dc9262d245a29d3da4595b11b6391d71041c"} Oct 30 22:24:14.273074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:14.272948 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podStartSLOduration=1.2729282259999999 podStartE2EDuration="1.272928226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:14.265766401 +0000 UTC m=+1782.234585967" watchObservedRunningTime="2026-10-30 22:24:14.272928226 +0000 UTC m=+1782.241747790" Oct 30 22:24:14.528977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:14.525372 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:15.527336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:15.527296 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:24:15.527832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:15.527371 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 22:24:16.525105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:16.525066 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:24:16.525285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:16.525135 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 22:24:17.525639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:17.525606 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:24:17.526028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:17.525656 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 22:24:18.525213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:18.525178 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:24:18.525390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:18.525243 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 22:24:19.113852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.113812 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:24:19.114989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.114969 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1df68f5-165d-4876-9379-27b2e703e508" containerName="istio-proxy" Oct 30 22:24:19.119317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.119289 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.217308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.217271 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.217757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.217697 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.217981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.217960 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9vj\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.218695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.218642 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.218924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.218897 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.219055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.218953 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.219128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.219113 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.219223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.219201 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.219553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.219527 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.246496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.246467 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-afe802-openshift-default-dockercfg-gt49w\"" Oct 30 22:24:19.250889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.250861 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:24:19.330410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.330330 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.330787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.330750 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331074 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331302 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331476 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9vj\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331610 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331695 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.331916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.331754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.332071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.332052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.332285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.332267 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.332702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.332301 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.333095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.333012 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.333771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.333743 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.333870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.332055 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.353685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.337362 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.374667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.374209 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.454617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.454378 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9vj\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.483343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.483303 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\") pod \"e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:19.556666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.556235 4292 patch_prober.go:29] interesting pod/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:24:19.556666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.556295 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 22:24:19.739036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:19.738965 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:20.056052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.056016 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:24:20.066027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.065994 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.110643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.110522 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:24:20.112752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.111807 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:24:20.141605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.140398 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-53cceb\"" Oct 30 22:24:20.141605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.140558 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-53cceb-dockercfg-crl7t\"" Oct 30 22:24:20.167809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.167756 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:24:20.202957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.202919 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:24:20.209564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.209537 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.237362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.237278 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:24:20.243218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.241981 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwjh\" (UniqueName: \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.243218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.242053 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.243218 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:20.242927 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f23de09_a8c2_44ce_99a2_f49ef3cb015a.slice/crio-24361cda54324855bafd12fc85addc8b9936bfed4d52a04fbfd845959e50c9a6 WatchSource:0}: Error finding container 24361cda54324855bafd12fc85addc8b9936bfed4d52a04fbfd845959e50c9a6: Status 404 returned error can't find the container with id 24361cda54324855bafd12fc85addc8b9936bfed4d52a04fbfd845959e50c9a6 Oct 30 22:24:20.245939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.245896 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:20.246056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.245983 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:20.246056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.246031 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:20.278664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.273068 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:24:20.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.349378 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnsq\" (UniqueName: \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.349522 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.349664 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwjh\" (UniqueName: \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.349735 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.352335 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.452668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.451153 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnsq\" (UniqueName: \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.452668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.451275 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.452668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.452307 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.465710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.462454 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwjh\" (UniqueName: \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\") pod \"payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.522366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.522332 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnsq\" (UniqueName: \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\") pod \"payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.533633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.533603 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:20.535176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.535155 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:20.536342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.536322 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:24:20.541628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.541608 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:20.712763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.712686 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:20.965207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:20.965181 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:24:21.168987 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:21.168954 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1b3c09_95a5_4a94_a674_77d7ddee3482.slice/crio-a0aaedc2a27270cb47ab020c9e92f4f02cbb8baf60bece3c71fbd39242108f65 WatchSource:0}: Error finding container a0aaedc2a27270cb47ab020c9e92f4f02cbb8baf60bece3c71fbd39242108f65: Status 404 returned error can't find the container with id a0aaedc2a27270cb47ab020c9e92f4f02cbb8baf60bece3c71fbd39242108f65 Oct 30 22:24:21.169436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.169415 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:24:21.205216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.205188 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" event={"ID":"9a1b3c09-95a5-4a94-a674-77d7ddee3482","Type":"ContainerStarted","Data":"a0aaedc2a27270cb47ab020c9e92f4f02cbb8baf60bece3c71fbd39242108f65"} Oct 30 22:24:21.207003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.206974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" event={"ID":"0cbbcbc0-31b4-4050-b556-c12e61da789b","Type":"ContainerStarted","Data":"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428"} Oct 30 22:24:21.207108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.207027 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" event={"ID":"0cbbcbc0-31b4-4050-b556-c12e61da789b","Type":"ContainerStarted","Data":"d229db41e7dfff8201bcfc1db15ed12f185d00611e6c91842f36fd5585d06d4b"} Oct 30 22:24:21.207108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.207058 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:21.208695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.208671 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerStarted","Data":"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963"} Oct 30 22:24:21.208762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.208719 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerStarted","Data":"24361cda54324855bafd12fc85addc8b9936bfed4d52a04fbfd845959e50c9a6"} Oct 30 22:24:21.333545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.333498 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" podStartSLOduration=1.333485721 podStartE2EDuration="1.333485721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:21.32937012 +0000 UTC m=+1789.298189687" watchObservedRunningTime="2026-10-30 22:24:21.333485721 +0000 UTC m=+1789.302305285" Oct 30 22:24:21.740412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.740359 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:21.747443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.747416 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:21.873885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:21.873829 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podStartSLOduration=2.8738158719999998 podStartE2EDuration="2.873815872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:21.46589734 +0000 UTC m=+1789.434716905" watchObservedRunningTime="2026-10-30 22:24:21.873815872 +0000 UTC m=+1789.842635435" Oct 30 22:24:22.219816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:22.219779 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" event={"ID":"9a1b3c09-95a5-4a94-a674-77d7ddee3482","Type":"ContainerStarted","Data":"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817"} Oct 30 22:24:22.220228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:22.219885 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:22.220312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:22.220274 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:22.221132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:22.221111 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:24:22.342687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:22.342636 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" podStartSLOduration=2.342622766 podStartE2EDuration="2.342622766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:22.339355553 +0000 UTC m=+1790.308175120" watchObservedRunningTime="2026-10-30 22:24:22.342622766 +0000 UTC m=+1790.311442349" Oct 30 22:24:23.514450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.514417 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:24:23.515001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.514620 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" containerID="cri-o://125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3" gracePeriod=30 Oct 30 22:24:23.567139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.567108 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:24:23.567364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.567340 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" containerID="cri-o://61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb" gracePeriod=30 Oct 30 22:24:23.572265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.572232 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:24:23.572470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.572443 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" containerID="cri-o://83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5" gracePeriod=30 Oct 30 22:24:23.615101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.615074 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:24:23.615256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:23.615238 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" containerID="cri-o://0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2" gracePeriod=30 Oct 30 22:24:24.071962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:24.071927 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:24.072128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:24.071999 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:26.641466 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:26.641431 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:24:26.838026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:26.837993 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:26.838171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:26.838052 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:28.890638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:28.890610 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:24:28.974759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:28.974732 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:24:29.015305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015275 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015328 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015535 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015656 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015709 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015719 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.015749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015743 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015777 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.015981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015935 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.016032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.015977 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\") pod \"669dbe44-e54c-46ef-8c21-0da05d534351\" (UID: \"669dbe44-e54c-46ef-8c21-0da05d534351\") " Oct 30 22:24:29.016210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016188 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.016382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016362 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.016382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016383 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.016960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016892 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669dbe44-e54c-46ef-8c21-0da05d534351-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.016960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016903 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.016960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016918 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.016960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016932 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.016960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.016942 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.018300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.018272 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:29.018450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.018434 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "kube-api-access-xvqnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.018761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.018737 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.018829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.018818 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy" pod "669dbe44-e54c-46ef-8c21-0da05d534351" (UID: "669dbe44-e54c-46ef-8c21-0da05d534351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.019057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.019041 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:24:29.022497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.022477 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:24:29.118464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118345 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.118464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118430 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.118464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118489 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118527 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118551 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118610 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118630 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118653 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118678 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118701 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118767 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.118820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118785 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118826 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118852 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118898 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.118934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119020 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119071 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119193 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119256 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119302 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119385 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119415 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\") pod \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\" (UID: \"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119446 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119488 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\") pod \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\" (UID: \"a6c0e3c1-d663-474b-9e22-f0e1183abd52\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.119595 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\") pod \"035687f1-5a2c-4283-94ac-72fb36ea801c\" (UID: \"035687f1-5a2c-4283-94ac-72fb36ea801c\") " Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.120524 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.120550 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669dbe44-e54c-46ef-8c21-0da05d534351-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.120567 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqnn\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-kube-api-access-xvqnn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.120598 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669dbe44-e54c-46ef-8c21-0da05d534351-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.120612 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669dbe44-e54c-46ef-8c21-0da05d534351-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.121091 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.121418 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.121662 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.121849 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.122009 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.122193 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.122841 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123031 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123183 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123283 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123503 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123549 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123701 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.123809 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.124819 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.126884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.126084 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.128873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.127215 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.129002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.128979 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.129133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129111 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "kube-api-access-kp2pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.129203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129183 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:29.129408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129371 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "kube-api-access-8xvfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.129498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129443 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:29.129498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129439 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo" pod "035687f1-5a2c-4283-94ac-72fb36ea801c" (UID: "035687f1-5a2c-4283-94ac-72fb36ea801c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:29.129646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129613 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:29.129646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129622 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "kube-api-access-79tlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.129770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.129727 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token" pod "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" (UID: "7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.130848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.130817 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token" pod "a6c0e3c1-d663-474b-9e22-f0e1183abd52" (UID: "a6c0e3c1-d663-474b-9e22-f0e1183abd52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.222740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222706 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222742 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tlh\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-kube-api-access-79tlh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222757 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222772 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222787 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222800 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222814 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222829 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222842 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222856 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035687f1-5a2c-4283-94ac-72fb36ea801c-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222869 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222883 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222896 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222911 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222925 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222938 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222960 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvfx\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-kube-api-access-8xvfx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222973 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2pr\" (UniqueName: \"kubernetes.io/projected/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-kube-api-access-kp2pr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.222988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.222987 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223000 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223015 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223027 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223039 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035687f1-5a2c-4283-94ac-72fb36ea801c-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223054 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223067 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c0e3c1-d663-474b-9e22-f0e1183abd52-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223079 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035687f1-5a2c-4283-94ac-72fb36ea801c-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.223897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.223093 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.276849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.276818 4292 generic.go:356] "Generic (PLEG): container finished" podID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerID="61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb" exitCode=0 Oct 30 22:24:29.277015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.276869 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" Oct 30 22:24:29.277015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.276896 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" event={"ID":"a6c0e3c1-d663-474b-9e22-f0e1183abd52","Type":"ContainerDied","Data":"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb"} Oct 30 22:24:29.277015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.276943 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf" event={"ID":"a6c0e3c1-d663-474b-9e22-f0e1183abd52","Type":"ContainerDied","Data":"c01ab042ba5a28e757be6ddaa2115744ecb777ffa141d77d52edaa80ec677672"} Oct 30 22:24:29.277015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.276971 4292 scope.go:122] "RemoveContainer" containerID="61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb" Oct 30 22:24:29.278667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.278637 4292 generic.go:356] "Generic (PLEG): container finished" podID="669dbe44-e54c-46ef-8c21-0da05d534351" containerID="125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3" exitCode=0 Oct 30 22:24:29.278818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.278799 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" Oct 30 22:24:29.280520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.280434 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" event={"ID":"669dbe44-e54c-46ef-8c21-0da05d534351","Type":"ContainerDied","Data":"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3"} Oct 30 22:24:29.280520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.280487 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4" event={"ID":"669dbe44-e54c-46ef-8c21-0da05d534351","Type":"ContainerDied","Data":"027d59de63717d903a5a052b8993e7a449f3c30876dcbeb9a148bc5d616cd449"} Oct 30 22:24:29.282403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.282381 4292 generic.go:356] "Generic (PLEG): container finished" podID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerID="83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5" exitCode=0 Oct 30 22:24:29.282627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.282562 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" event={"ID":"035687f1-5a2c-4283-94ac-72fb36ea801c","Type":"ContainerDied","Data":"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5"} Oct 30 22:24:29.282712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.282645 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" event={"ID":"035687f1-5a2c-4283-94ac-72fb36ea801c","Type":"ContainerDied","Data":"a7ea2e0fb9f2638df6b68413d764048fb2176c279f5a2154c0681f8baaf617f6"} Oct 30 22:24:29.282783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.282716 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8" Oct 30 22:24:29.286619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.285010 4292 generic.go:356] "Generic (PLEG): container finished" podID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerID="0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2" exitCode=0 Oct 30 22:24:29.286619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.285102 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerDied","Data":"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2"} Oct 30 22:24:29.286619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.285141 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" event={"ID":"7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321","Type":"ContainerDied","Data":"44098fa21e14e4f97e4bc4ae6a9a752b8027ef0e2bac463a036c3770069d88bf"} Oct 30 22:24:29.286619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.285215 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79" Oct 30 22:24:29.298181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.298160 4292 scope.go:122] "RemoveContainer" containerID="61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb" Oct 30 22:24:29.298505 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:29.298482 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb\": container with ID starting with 61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb not found: ID does not exist" containerID="61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb" Oct 30 22:24:29.298588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.298519 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb"} err="failed to get container status \"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb\": rpc error: code = NotFound desc = could not find container \"61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb\": container with ID starting with 61a2b032bc01ae7a256c467de9aaf2862c7e3d8347742366b8660c2f8716bbbb not found: ID does not exist" Oct 30 22:24:29.298588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.298542 4292 scope.go:122] "RemoveContainer" containerID="125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3" Oct 30 22:24:29.315101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.315003 4292 scope.go:122] "RemoveContainer" containerID="125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3" Oct 30 22:24:29.315550 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:29.315420 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3\": container with ID starting with 125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3 not found: ID does not exist" containerID="125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3" Oct 30 22:24:29.315550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.315451 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3"} err="failed to get container status \"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3\": rpc error: code = NotFound desc = could not find container \"125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3\": container with ID starting with 125f2f6b8b4721a8f0b5a6ab79eab6be4ddda0782e48211317cac11e49ee5bb3 not found: ID does not exist" Oct 30 22:24:29.315550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.315474 4292 scope.go:122] "RemoveContainer" containerID="83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5" Oct 30 22:24:29.331960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.331940 4292 scope.go:122] "RemoveContainer" containerID="83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5" Oct 30 22:24:29.332293 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:29.332273 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5\": container with ID starting with 83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5 not found: ID does not exist" containerID="83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5" Oct 30 22:24:29.332379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.332303 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5"} err="failed to get container status \"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5\": rpc error: code = NotFound desc = could not find container \"83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5\": container with ID starting with 83a349e207a485178d9aca4e5e488137a95a9e4acb16ac354dea62a6e56d22d5 not found: ID does not exist" Oct 30 22:24:29.332379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.332330 4292 scope.go:122] "RemoveContainer" containerID="0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2" Oct 30 22:24:29.343034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.343016 4292 scope.go:122] "RemoveContainer" containerID="ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013" Oct 30 22:24:29.353768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.353732 4292 scope.go:122] "RemoveContainer" containerID="0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2" Oct 30 22:24:29.354070 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:29.354044 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2\": container with ID starting with 0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2 not found: ID does not exist" containerID="0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2" Oct 30 22:24:29.354128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.354078 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2"} err="failed to get container status \"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2\": rpc error: code = NotFound desc = could not find container \"0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2\": container with ID starting with 0460f3f2591ae1570a9a81dcf41b99e1ad4c6b774febce5311587059324900c2 not found: ID does not exist" Oct 30 22:24:29.354128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.354098 4292 scope.go:122] "RemoveContainer" containerID="ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013" Oct 30 22:24:29.354359 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:29.354333 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013\": container with ID starting with ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013 not found: ID does not exist" containerID="ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013" Oct 30 22:24:29.354408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.354364 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013"} err="failed to get container status \"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013\": rpc error: code = NotFound desc = could not find container \"ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013\": container with ID starting with ac59a01f9382a387921360c8bc06a218bbf5d7b98901d1cad79d2a346bee6013 not found: ID does not exist" Oct 30 22:24:29.598308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.598278 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:24:29.603198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.603167 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-zvsj4"] Oct 30 22:24:29.609310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.609283 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:24:29.610342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610322 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.610391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610347 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.610391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610371 4292 cpu_manager.go:414] "containerMap: removing container" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" Oct 30 22:24:29.610391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610379 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" Oct 30 22:24:29.610490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610412 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" Oct 30 22:24:29.610490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610421 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" Oct 30 22:24:29.610490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610440 4292 cpu_manager.go:414] "containerMap: removing container" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" Oct 30 22:24:29.610490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610449 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" Oct 30 22:24:29.610640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610626 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.610675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610647 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" containerName="istio-proxy" Oct 30 22:24:29.610675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610660 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.610675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610670 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" containerName="istio-proxy" Oct 30 22:24:29.610770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.610680 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" containerName="istio-proxy" Oct 30 22:24:29.616783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.616758 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.664790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.664750 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:24:29.673671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.673602 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-afe802-dockercfg-nmjbx\"" Oct 30 22:24:29.673934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.673721 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-afe802\"" Oct 30 22:24:29.716452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.716386 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:24:29.731247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.731178 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.731645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.731555 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglmc\" (UniqueName: \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.746445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.746415 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-jhx79"] Oct 30 22:24:29.751880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.751851 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:24:29.753184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.753155 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.753344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.753326 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" containerName="istio-proxy" Oct 30 22:24:29.762512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.762491 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:29.803753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.803655 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:24:29.835698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.835662 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:24:29.836860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.836825 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglmc\" (UniqueName: \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.837463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.837001 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.838522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.838491 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.838678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.838554 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-c8sqf"] Oct 30 22:24:29.922408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.921179 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglmc\" (UniqueName: \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\") pod \"payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.933036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.933002 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:29.939005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.938974 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7rr\" (UniqueName: \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:29.939135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.939053 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:29.956657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.956629 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:24:29.962601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:29.962563 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-97aa64-openshift-default-569955745-9zrs8"] Oct 30 22:24:30.041015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.040972 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7rr\" (UniqueName: \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:30.041171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.041034 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:30.041805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.041779 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:30.105015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.104938 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7rr\" (UniqueName: \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\") pod \"payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:30.148944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.148914 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035687f1-5a2c-4283-94ac-72fb36ea801c" path="/var/lib/kubelet/pods/035687f1-5a2c-4283-94ac-72fb36ea801c/volumes" Oct 30 22:24:30.149417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.149403 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669dbe44-e54c-46ef-8c21-0da05d534351" path="/var/lib/kubelet/pods/669dbe44-e54c-46ef-8c21-0da05d534351/volumes" Oct 30 22:24:30.149874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.149860 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321" path="/var/lib/kubelet/pods/7bdd9e96-cc13-4fcd-9e95-0a3cfa08a321/volumes" Oct 30 22:24:30.150657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.150642 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c0e3c1-d663-474b-9e22-f0e1183abd52" path="/var/lib/kubelet/pods/a6c0e3c1-d663-474b-9e22-f0e1183abd52/volumes" Oct 30 22:24:30.322797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.322769 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:24:30.322938 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:30.322912 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d835a0_2434_4eda_927e_10b5a984c28c.slice/crio-665612cba9a415e0f14503a3ceefca41d70072868c93c31b721ebf37b51c13ec WatchSource:0}: Error finding container 665612cba9a415e0f14503a3ceefca41d70072868c93c31b721ebf37b51c13ec: Status 404 returned error can't find the container with id 665612cba9a415e0f14503a3ceefca41d70072868c93c31b721ebf37b51c13ec Oct 30 22:24:30.384569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.384541 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:30.765922 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:30.765892 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8acacac_90b2_4685_b150_844de8d2aed8.slice/crio-21bb403ea12ff8d8fb68b3473edcad6567d6d662c4977bd8381f69ee053c437d WatchSource:0}: Error finding container 21bb403ea12ff8d8fb68b3473edcad6567d6d662c4977bd8381f69ee053c437d: Status 404 returned error can't find the container with id 21bb403ea12ff8d8fb68b3473edcad6567d6d662c4977bd8381f69ee053c437d Oct 30 22:24:30.767196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:30.767164 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:24:31.305911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.305869 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" event={"ID":"80d835a0-2434-4eda-927e-10b5a984c28c","Type":"ContainerStarted","Data":"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2"} Oct 30 22:24:31.306306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.305940 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" event={"ID":"80d835a0-2434-4eda-927e-10b5a984c28c","Type":"ContainerStarted","Data":"665612cba9a415e0f14503a3ceefca41d70072868c93c31b721ebf37b51c13ec"} Oct 30 22:24:31.306306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.305966 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:31.307548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.307520 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" event={"ID":"b8acacac-90b2-4685-b150-844de8d2aed8","Type":"ContainerStarted","Data":"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab"} Oct 30 22:24:31.307687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.307569 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" event={"ID":"b8acacac-90b2-4685-b150-844de8d2aed8","Type":"ContainerStarted","Data":"21bb403ea12ff8d8fb68b3473edcad6567d6d662c4977bd8381f69ee053c437d"} Oct 30 22:24:31.307687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.307638 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:31.431847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.431801 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" podStartSLOduration=2.431787665 podStartE2EDuration="2.431787665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:31.425374929 +0000 UTC m=+1799.394194504" watchObservedRunningTime="2026-10-30 22:24:31.431787665 +0000 UTC m=+1799.400607228" Oct 30 22:24:31.550114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:31.550053 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" podStartSLOduration=2.550033754 podStartE2EDuration="2.550033754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:31.549052033 +0000 UTC m=+1799.517871597" watchObservedRunningTime="2026-10-30 22:24:31.550033754 +0000 UTC m=+1799.518853319" Oct 30 22:24:32.223734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:32.223706 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:24:33.231644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:33.231615 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:24:36.694273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.694242 4292 scope.go:122] "RemoveContainer" containerID="d11a6314cd1cfaace331e8482441353f62d7edf3cfcdf773e0c3c7b167ad8b80" Oct 30 22:24:36.705813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.705792 4292 scope.go:122] "RemoveContainer" containerID="7a4e661b98fe0e4e72a3116c18d9507e7f5e302eb463c7de2b42073dcdcb61c7" Oct 30 22:24:36.715472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.715442 4292 scope.go:122] "RemoveContainer" containerID="8a7bfdc98cbeba4e3f6d962262afb184e9b4e678e7ed6968f77ea8142af4d581" Oct 30 22:24:36.725254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.725238 4292 scope.go:122] "RemoveContainer" containerID="e1ccd44a7fdd264f2ec700febf45e0b3d18795fe5c57e0d75f879d6bf8ea3e3e" Oct 30 22:24:36.735284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.735260 4292 scope.go:122] "RemoveContainer" containerID="53f9406835410d30b5ced3b8c36b3e465ba6a3f65c1b2eb3ad742ff488b0e414" Oct 30 22:24:36.744184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.744166 4292 scope.go:122] "RemoveContainer" containerID="a1a691110c0304bd52208fc95d478455bde97fe56c7c9f10baefa6d4afd82d7e" Oct 30 22:24:36.752931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.752914 4292 scope.go:122] "RemoveContainer" containerID="0a30f6576a42f934140736cf217af57b602322bba70c7add156b0c20f4092e58" Oct 30 22:24:36.762125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.762108 4292 scope.go:122] "RemoveContainer" containerID="1594204f4d71ef1bdab28873a5796e98ff9975b8047494e3f2e8da0097b8d868" Oct 30 22:24:36.770912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.770890 4292 scope.go:122] "RemoveContainer" containerID="1d01239ecff3298c08a48303b94bd203fa4ad34241a13e094ddf70cc3c5053b0" Oct 30 22:24:36.779312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:36.779298 4292 scope.go:122] "RemoveContainer" containerID="d9b8177d7033d96a5b57334ee7c10f53aa2803110b85fc413b69fc2e0c994a91" Oct 30 22:24:42.318770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:42.318738 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:24:42.319168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:42.318812 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:24:55.125314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.125281 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:24:55.133273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.133250 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.193958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.193911 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:24:55.271594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.243640 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.271594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.243896 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wws\" (UniqueName: \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.271594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.244428 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a42692-dockercfg-b7n6d\"" Oct 30 22:24:55.271594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.244514 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a42692\"" Oct 30 22:24:55.337781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.337740 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:24:55.351324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.348823 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.351324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.349930 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wws\" (UniqueName: \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.351324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.351249 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.353792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.353721 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.355005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.354987 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:24:55.420999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.420926 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wws\" (UniqueName: \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\") pod \"payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.447828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.447797 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:55.451496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.451462 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72kxx\" (UniqueName: \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.451723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.451701 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.553020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.552977 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.553968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.553214 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72kxx\" (UniqueName: \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.554296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.554272 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.624697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.624664 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72kxx\" (UniqueName: \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\") pod \"payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.681913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.681833 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:55.841566 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:55.841472 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9924d8_7c90_458a_933b_98037875436d.slice/crio-c828c6c6f5937c2ffd212482930c5caef56079c1f31eded3397a6b11b8fd5cec WatchSource:0}: Error finding container c828c6c6f5937c2ffd212482930c5caef56079c1f31eded3397a6b11b8fd5cec: Status 404 returned error can't find the container with id c828c6c6f5937c2ffd212482930c5caef56079c1f31eded3397a6b11b8fd5cec Oct 30 22:24:55.842771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:55.842682 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:24:56.077030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.076999 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:24:56.077588 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:24:56.077547 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8b4344_68fd_41b5_b339_6bb048428101.slice/crio-ec5f8f42036f301c7082e4ceeb8a8b7e48a5743d9b9d1335bcb8bd04bdf03a59 WatchSource:0}: Error finding container ec5f8f42036f301c7082e4ceeb8a8b7e48a5743d9b9d1335bcb8bd04bdf03a59: Status 404 returned error can't find the container with id ec5f8f42036f301c7082e4ceeb8a8b7e48a5743d9b9d1335bcb8bd04bdf03a59 Oct 30 22:24:56.510998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.510960 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" event={"ID":"da8b4344-68fd-41b5-b339-6bb048428101","Type":"ContainerStarted","Data":"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9"} Oct 30 22:24:56.511402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.511029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" event={"ID":"da8b4344-68fd-41b5-b339-6bb048428101","Type":"ContainerStarted","Data":"ec5f8f42036f301c7082e4ceeb8a8b7e48a5743d9b9d1335bcb8bd04bdf03a59"} Oct 30 22:24:56.511402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.511062 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:24:56.512810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.512780 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" event={"ID":"3d9924d8-7c90-458a-933b-98037875436d","Type":"ContainerStarted","Data":"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73"} Oct 30 22:24:56.512927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.512845 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:24:56.512927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.512870 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" event={"ID":"3d9924d8-7c90-458a-933b-98037875436d","Type":"ContainerStarted","Data":"c828c6c6f5937c2ffd212482930c5caef56079c1f31eded3397a6b11b8fd5cec"} Oct 30 22:24:56.639355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.639305 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" podStartSLOduration=1.639290095 podStartE2EDuration="1.639290095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:56.63694835 +0000 UTC m=+1824.605767915" watchObservedRunningTime="2026-10-30 22:24:56.639290095 +0000 UTC m=+1824.608109692" Oct 30 22:24:56.646618 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:24:56.646552 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:24:56.765837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:56.765746 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" podStartSLOduration=1.76573418 podStartE2EDuration="1.76573418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:56.762784062 +0000 UTC m=+1824.731603630" watchObservedRunningTime="2026-10-30 22:24:56.76573418 +0000 UTC m=+1824.734553744" Oct 30 22:24:58.529463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.529429 4292 generic.go:356] "Generic (PLEG): container finished" podID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerID="6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f" exitCode=0 Oct 30 22:24:58.530005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.529503 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerDied","Data":"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f"} Oct 30 22:24:58.530005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.529922 4292 scope.go:122] "RemoveContainer" containerID="6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f" Oct 30 22:24:58.531606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.531561 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:58.531712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.531646 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:58.531712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:58.531682 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:59.551745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:24:59.551706 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerStarted","Data":"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2"} Oct 30 22:25:00.273820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:00.273785 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:25:00.273996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:00.273958 4292 patch_prober.go:29] interesting pod/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.274049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:00.274001 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:25:01.274709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:01.274669 4292 patch_prober.go:29] interesting pod/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:25:01.275216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:01.274731 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:25:02.274409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:02.274370 4292 patch_prober.go:29] interesting pod/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:25:02.274635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:02.274428 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:25:03.106600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:03.085393 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx"] Oct 30 22:25:03.134805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:03.134765 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334aaa76-m5dkx"] Oct 30 22:25:03.273992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:03.273957 4292 patch_prober.go:29] interesting pod/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:25:03.274163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:03.274004 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:25:04.150313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:04.150278 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9696683c-bc35-48a7-b03f-763f52cbf872" path="/var/lib/kubelet/pods/9696683c-bc35-48a7-b03f-763f52cbf872/volumes" Oct 30 22:25:04.274570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:04.274533 4292 patch_prober.go:29] interesting pod/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:25:04.274787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:04.274644 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:25:05.279165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:05.279133 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:25:05.279619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:05.279599 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:25:05.280242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:05.280222 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:25:07.524437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:07.524405 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:25:07.524930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:07.524548 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:25:26.643632 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:25:26.643593 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:25:37.050112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.050074 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8"] Oct 30 22:25:37.055861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.055841 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.111746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.111726 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:37.143316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.143281 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.143469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.143433 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.143541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.143527 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxk6d\" (UniqueName: \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.161442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.161416 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:37.161442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.161431 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:37.161712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.161694 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:25:37.166067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.166047 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8"] Oct 30 22:25:37.244632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.244602 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.244766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.244717 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.244830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.244798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxk6d\" (UniqueName: \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.245488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.245463 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.247206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.247183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.309509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.309454 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxk6d\" (UniqueName: \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\") pod \"maas-api-revoke-keys-e2e-mt-367f4491-f5bl8\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.370362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.370337 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:37.756178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.756146 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8"] Oct 30 22:25:37.756866 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:25:37.756835 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac05f11_9794_457b_9a35_efa4c2ff61b2.slice/crio-0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c WatchSource:0}: Error finding container 0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c: Status 404 returned error can't find the container with id 0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c Oct 30 22:25:37.990767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:37.990730 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" event={"ID":"dac05f11-9794-457b-9a35-efa4c2ff61b2","Type":"ContainerStarted","Data":"0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c"} Oct 30 22:25:39.000079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:38.999977 4292 generic.go:356] "Generic (PLEG): container finished" podID="dac05f11-9794-457b-9a35-efa4c2ff61b2" containerID="659d9c7f2f7a3890ecaa3500a97df09d3ebe0a70271e68fc056442db496c27c4" exitCode=0 Oct 30 22:25:39.000553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:39.000068 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" event={"ID":"dac05f11-9794-457b-9a35-efa4c2ff61b2","Type":"ContainerDied","Data":"659d9c7f2f7a3890ecaa3500a97df09d3ebe0a70271e68fc056442db496c27c4"} Oct 30 22:25:40.198986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.198957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:40.288237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.288170 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\") pod \"dac05f11-9794-457b-9a35-efa4c2ff61b2\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " Oct 30 22:25:40.288367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.288269 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\" (UniqueName: \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\") pod \"dac05f11-9794-457b-9a35-efa4c2ff61b2\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " Oct 30 22:25:40.288467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.288453 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\") pod \"dac05f11-9794-457b-9a35-efa4c2ff61b2\" (UID: \"dac05f11-9794-457b-9a35-efa4c2ff61b2\") " Oct 30 22:25:40.288793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.288771 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca" pod "dac05f11-9794-457b-9a35-efa4c2ff61b2" (UID: "dac05f11-9794-457b-9a35-efa4c2ff61b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:40.289073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.289057 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.290266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.290244 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d" pod "dac05f11-9794-457b-9a35-efa4c2ff61b2" (UID: "dac05f11-9794-457b-9a35-efa4c2ff61b2"). InnerVolumeSpecName "kube-api-access-fxk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:40.290315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.290298 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca" pod "dac05f11-9794-457b-9a35-efa4c2ff61b2" (UID: "dac05f11-9794-457b-9a35-efa4c2ff61b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:40.390543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.390510 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac05f11-9794-457b-9a35-efa4c2ff61b2-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.390543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:40.390540 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxk6d\" (UniqueName: \"kubernetes.io/projected/dac05f11-9794-457b-9a35-efa4c2ff61b2-kube-api-access-fxk6d\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:41.016868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:41.016839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" Oct 30 22:25:41.017042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:41.016870 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8" event={"ID":"dac05f11-9794-457b-9a35-efa4c2ff61b2","Type":"ContainerDied","Data":"0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c"} Oct 30 22:25:41.017042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:41.016918 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b51d6000646a130e967c96faf2292324cffc08a1c8cccbb3d4c8a1b23651c9c" Oct 30 22:25:48.583433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:48.583398 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:25:53.584005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:53.583949 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" podUID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" containerName="payload-processing" containerID="cri-o://5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a" gracePeriod=25 Oct 30 22:25:54.046047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.046015 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:25:54.136844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.136807 4292 generic.go:356] "Generic (PLEG): container finished" podID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" containerID="5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a" exitCode=0 Oct 30 22:25:54.137129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.136889 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" Oct 30 22:25:54.137385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.136912 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" event={"ID":"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87","Type":"ContainerDied","Data":"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a"} Oct 30 22:25:54.137535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.137516 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn" event={"ID":"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87","Type":"ContainerDied","Data":"c41190bef500eb4c2ae4075e420f9d72579a6302bd56b55ccfae8861201a98c5"} Oct 30 22:25:54.137684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.137669 4292 scope.go:122] "RemoveContainer" containerID="5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a" Oct 30 22:25:54.143163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.143140 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\" (UniqueName: \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\") pod \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " Oct 30 22:25:54.144916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.144878 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\") pod \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\" (UID: \"e1447dc5-b8a2-4d1c-8283-e8951c7bfe87\") " Oct 30 22:25:54.149488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.149464 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume" pod "e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" (UID: "e1447dc5-b8a2-4d1c-8283-e8951c7bfe87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:54.154446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.154406 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k" pod "e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" (UID: "e1447dc5-b8a2-4d1c-8283-e8951c7bfe87"). InnerVolumeSpecName "kube-api-access-7ss4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.193480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.193455 4292 scope.go:122] "RemoveContainer" containerID="5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a" Oct 30 22:25:54.194067 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:25:54.194032 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a\": container with ID starting with 5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a not found: ID does not exist" containerID="5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a" Oct 30 22:25:54.194186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.194078 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a"} err="failed to get container status \"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a\": rpc error: code = NotFound desc = could not find container \"5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a\": container with ID starting with 5f3ad045805df679006895a9316bd83837e1057adcfa745ff3d75943d746859a not found: ID does not exist" Oct 30 22:25:54.248688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.248608 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ss4k\" (UniqueName: \"kubernetes.io/projected/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-kube-api-access-7ss4k\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.248688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.248646 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.582004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.581968 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:25:54.629274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:54.629240 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-367f4491-7c7684555f-jvrrn"] Oct 30 22:25:56.150079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:25:56.150038 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" path="/var/lib/kubelet/pods/e1447dc5-b8a2-4d1c-8283-e8951c7bfe87/volumes" Oct 30 22:25:56.651116 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:25:56.651080 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:26:23.733604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:23.733556 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:26:23.734045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:23.733801 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" containerID="cri-o://7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313" gracePeriod=30 Oct 30 22:26:24.831626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:24.831588 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:26:24.832131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:24.831788 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" containerID="cri-o://9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc" gracePeriod=30 Oct 30 22:26:26.643507 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:26.643456 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:26:27.416848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.416814 4292 generic.go:356] "Generic (PLEG): container finished" podID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerID="13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963" exitCode=0 Oct 30 22:26:27.417087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.416891 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerDied","Data":"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963"} Oct 30 22:26:27.417359 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.417342 4292 scope.go:122] "RemoveContainer" containerID="13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963" Oct 30 22:26:27.418947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.418915 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:27.419028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.418992 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:27.419066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:27.419030 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:28.427050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:28.427014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerStarted","Data":"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545"} Oct 30 22:26:28.739422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:28.739337 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:26:28.739606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:28.739446 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:26:28.739606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:28.739501 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 22:26:29.073019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.072995 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:26:29.127740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.127707 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.127890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.127772 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.127890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.127836 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.127890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.127864 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128000 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.127990 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:29.128161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128061 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:29.128161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128133 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128210 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128291 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\") pod \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\" (UID: \"c5d6e81a-de8c-45ed-be7b-93c2bacb3235\") " Oct 30 22:26:29.128378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128369 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:29.128505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128476 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:29.128791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128761 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:29.129014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.128994 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.129063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.129023 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.129063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.129039 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.129063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.129053 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.129162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.129068 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.130216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.130194 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "kube-api-access-5gbsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:29.130330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.130253 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:29.130547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.130531 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:29.130633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.130605 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo" pod "c5d6e81a-de8c-45ed-be7b-93c2bacb3235" (UID: "c5d6e81a-de8c-45ed-be7b-93c2bacb3235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:29.230671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.230635 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.230671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.230670 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.230883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.230687 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.230883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.230701 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gbsl\" (UniqueName: \"kubernetes.io/projected/c5d6e81a-de8c-45ed-be7b-93c2bacb3235-kube-api-access-5gbsl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.438559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.438520 4292 generic.go:356] "Generic (PLEG): container finished" podID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerID="7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313" exitCode=0 Oct 30 22:26:29.438559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.438595 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" Oct 30 22:26:29.439176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.438617 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" event={"ID":"c5d6e81a-de8c-45ed-be7b-93c2bacb3235","Type":"ContainerDied","Data":"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313"} Oct 30 22:26:29.439176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.438677 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr" event={"ID":"c5d6e81a-de8c-45ed-be7b-93c2bacb3235","Type":"ContainerDied","Data":"f3eb354ee25fb43a38f21f63267af7738a04ff82ff9ccde9d3b40f9361475831"} Oct 30 22:26:29.439176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.438703 4292 scope.go:122] "RemoveContainer" containerID="7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313" Oct 30 22:26:29.461941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.461920 4292 scope.go:122] "RemoveContainer" containerID="7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313" Oct 30 22:26:29.462232 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:29.462210 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313\": container with ID starting with 7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313 not found: ID does not exist" containerID="7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313" Oct 30 22:26:29.462327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.462244 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313"} err="failed to get container status \"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313\": rpc error: code = NotFound desc = could not find container \"7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313\": container with ID starting with 7af847568e74179666202d49ff596fe49189a730b565ed5e96b8aa242409a313 not found: ID does not exist" Oct 30 22:26:29.636534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.636489 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:26:29.714648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.714293 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-gfzhr"] Oct 30 22:26:29.740134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.740096 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:26:29.740287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.740175 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 22:26:29.911596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.911537 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:26:29.911770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:29.911627 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 22:26:30.151251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.151221 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" path="/var/lib/kubelet/pods/c5d6e81a-de8c-45ed-be7b-93c2bacb3235/volumes" Oct 30 22:26:30.168449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.168423 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:26:30.245765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.245730 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.245930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.245886 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.245982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.245934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.245982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.245954 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.245982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.245973 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.246143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246040 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.246143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246066 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.246143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246100 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.246143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246133 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\") pod \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\" (UID: \"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267\") " Oct 30 22:26:30.246354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246244 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:30.246354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246339 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:30.246450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246435 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:30.246721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:30.247612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.246807 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:30.247612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.247464 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.247612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.247485 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.247612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.247502 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.247612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.247517 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.248278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.248254 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:30.248556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.248532 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:30.248698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.248558 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "kube-api-access-qzmbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:30.248698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.248601 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo" pod "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" (UID: "dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:30.349406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.349371 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.349406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.349402 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzmbz\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-kube-api-access-qzmbz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.349406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.349412 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.349743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.349420 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.349743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.349429 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:30.448165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.448127 4292 generic.go:356] "Generic (PLEG): container finished" podID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerID="9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc" exitCode=0 Oct 30 22:26:30.448662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.448184 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" Oct 30 22:26:30.448662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.448203 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" event={"ID":"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267","Type":"ContainerDied","Data":"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc"} Oct 30 22:26:30.448662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.448262 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr" event={"ID":"dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267","Type":"ContainerDied","Data":"0695068ec6a81884cd234b85c0c86e5614a13bb62aedbbe06b8eb00218cf08c9"} Oct 30 22:26:30.448662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.448294 4292 scope.go:122] "RemoveContainer" containerID="9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc" Oct 30 22:26:30.460694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.460673 4292 scope.go:122] "RemoveContainer" containerID="9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc" Oct 30 22:26:30.460926 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:30.460908 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc\": container with ID starting with 9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc not found: ID does not exist" containerID="9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc" Oct 30 22:26:30.460979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.460933 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc"} err="failed to get container status \"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc\": rpc error: code = NotFound desc = could not find container \"9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc\": container with ID starting with 9292e9a6fab811f3719ed94665743319af344e2e69f35f808c8463fef42b45cc not found: ID does not exist" Oct 30 22:26:30.624401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.624311 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:26:30.629107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.629081 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-2k7pr"] Oct 30 22:26:30.740391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.740357 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:26:30.740559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:30.740407 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 22:26:31.740143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:31.740110 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:26:31.740524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:31.740166 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 22:26:32.219964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:32.218968 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" path="/var/lib/kubelet/pods/dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267/volumes" Oct 30 22:26:32.739970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:32.739933 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:26:32.740173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:32.740002 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 22:26:33.744770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:33.744738 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:26:33.745229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:33.745153 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:26:33.745731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:33.745713 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:26:36.326247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.326208 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:26:36.327235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327210 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" Oct 30 22:26:36.327235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327234 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327273 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327280 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327304 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" containerName="payload-processing" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327312 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" containerName="payload-processing" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327324 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dac05f11-9794-457b-9a35-efa4c2ff61b2" containerName="revoke-keys" Oct 30 22:26:36.327398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327332 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac05f11-9794-457b-9a35-efa4c2ff61b2" containerName="revoke-keys" Oct 30 22:26:36.327750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327501 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1447dc5-b8a2-4d1c-8283-e8951c7bfe87" containerName="payload-processing" Oct 30 22:26:36.327750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327513 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac05f11-9794-457b-9a35-efa4c2ff61b2" containerName="revoke-keys" Oct 30 22:26:36.327750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327532 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcacb8fd-a8b2-4a1f-8ef1-c72d2043f267" containerName="istio-proxy" Oct 30 22:26:36.327750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.327544 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d6e81a-de8c-45ed-be7b-93c2bacb3235" containerName="istio-proxy" Oct 30 22:26:36.332962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.332924 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.374873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.374757 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.381415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.381385 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.384210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.384183 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.384402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.384387 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.384623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.384607 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.384873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.384842 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.384992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.384976 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.385106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.385091 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8gj\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.385293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.385280 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.391307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.391284 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7fd7ba-openshift-default-dockercfg-5txxt\"" Oct 30 22:26:36.456057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.456017 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:26:36.491409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.491362 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.491655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.491452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492029 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492202 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492268 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492339 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492505 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.492658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.492554 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8gj\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.496890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.496862 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.497326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.497305 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.498057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.498036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.499078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.499057 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.499475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.499442 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.515281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.515249 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.516985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.516954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.561458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.561425 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.561654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.561627 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8gj\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\") pod \"e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.697168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.697071 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:36.918736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.918697 4292 scope.go:122] "RemoveContainer" containerID="518fe94b2d374f3230eae4102c721e749babf38281f3057b690529e4537f19de" Oct 30 22:26:36.928719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.928685 4292 scope.go:122] "RemoveContainer" containerID="72b7f8f541ee06a459b6df38585bd030f2e827cfe379d3d77efac9b153bf9b90" Oct 30 22:26:36.941492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.941466 4292 scope.go:122] "RemoveContainer" containerID="a56da57c041f93c88bf5af577fa5251d26680ab6e264c4b36b3ccb21e7547441" Oct 30 22:26:36.950492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:36.950468 4292 scope.go:122] "RemoveContainer" containerID="1630fefefe26f2a65e9fdc28f969163ee7364bd8f18d16010fa491e0aa3ebebc" Oct 30 22:26:37.146006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.145972 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:26:37.180385 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:26:37.180356 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b14f4ec_bdc4_48c8_b06e_dd1a3f3efc72.slice/crio-00735abe1760ef5329bd2bfc842089d18eadea428d24bd03b6029c25a228457a WatchSource:0}: Error finding container 00735abe1760ef5329bd2bfc842089d18eadea428d24bd03b6029c25a228457a: Status 404 returned error can't find the container with id 00735abe1760ef5329bd2bfc842089d18eadea428d24bd03b6029c25a228457a Oct 30 22:26:37.183039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.183005 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:37.183118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.183075 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:37.183118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.183107 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:37.531793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.531702 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerStarted","Data":"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5"} Oct 30 22:26:37.532158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.531804 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerStarted","Data":"00735abe1760ef5329bd2bfc842089d18eadea428d24bd03b6029c25a228457a"} Oct 30 22:26:37.659989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.659928 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podStartSLOduration=1.6599134979999999 podStartE2EDuration="1.659913498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:37.656949395 +0000 UTC m=+1925.625768958" watchObservedRunningTime="2026-10-30 22:26:37.659913498 +0000 UTC m=+1925.628733062" Oct 30 22:26:37.697667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:37.697638 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:38.704148 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:38.704119 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:39.546819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:39.546784 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:39.547727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:39.547709 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:26:44.407645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.407612 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:26:44.410857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.410832 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.465386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.465361 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-7fd7ba\"" Oct 30 22:26:44.557594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.552464 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-7fd7ba-dockercfg-zkmlw\"" Oct 30 22:26:44.557594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.553949 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz5q\" (UniqueName: \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.557594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.554051 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.601845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.601804 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:26:44.631043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.631005 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:26:44.637980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.637952 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.704773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.702991 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.704773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.703221 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwdc\" (UniqueName: \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.704773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.703417 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz5q\" (UniqueName: \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.704773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.703527 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.704773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.704494 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.748653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.748103 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:26:44.769615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.769558 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz5q\" (UniqueName: \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\") pod \"payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:44.805075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.805039 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.805075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.805076 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwdc\" (UniqueName: \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.805904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.805881 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:44.875770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:44.875736 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwdc\" (UniqueName: \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\") pod \"payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:45.026150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.026033 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:45.030067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.030039 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:45.413700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.413669 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:26:45.414435 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:26:45.414410 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06d7b86_75a3_4344_9b32_9bc1dc14fc22.slice/crio-50da2b6fbbac9de94f45abd13502a8c87317b1dbdc6b5c47714472cdae043ed7 WatchSource:0}: Error finding container 50da2b6fbbac9de94f45abd13502a8c87317b1dbdc6b5c47714472cdae043ed7: Status 404 returned error can't find the container with id 50da2b6fbbac9de94f45abd13502a8c87317b1dbdc6b5c47714472cdae043ed7 Oct 30 22:26:45.462073 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:26:45.462044 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e36320_98ed_4282_9a83_666aa7856d93.slice/crio-446f0300f87a7ba39a7cb83c82f0c543fdaf28ff41eb6a3399eeacb0af785249 WatchSource:0}: Error finding container 446f0300f87a7ba39a7cb83c82f0c543fdaf28ff41eb6a3399eeacb0af785249: Status 404 returned error can't find the container with id 446f0300f87a7ba39a7cb83c82f0c543fdaf28ff41eb6a3399eeacb0af785249 Oct 30 22:26:45.463384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.463351 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:26:45.694453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.694345 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" event={"ID":"99e36320-98ed-4282-9a83-666aa7856d93","Type":"ContainerStarted","Data":"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411"} Oct 30 22:26:45.694453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.694419 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:45.694453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.694440 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" event={"ID":"99e36320-98ed-4282-9a83-666aa7856d93","Type":"ContainerStarted","Data":"446f0300f87a7ba39a7cb83c82f0c543fdaf28ff41eb6a3399eeacb0af785249"} Oct 30 22:26:45.696268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.696128 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" event={"ID":"d06d7b86-75a3-4344-9b32-9bc1dc14fc22","Type":"ContainerStarted","Data":"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74"} Oct 30 22:26:45.696268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.696181 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" event={"ID":"d06d7b86-75a3-4344-9b32-9bc1dc14fc22","Type":"ContainerStarted","Data":"50da2b6fbbac9de94f45abd13502a8c87317b1dbdc6b5c47714472cdae043ed7"} Oct 30 22:26:45.696912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.696881 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:45.826950 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.824232 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" podStartSLOduration=1.8242095969999998 podStartE2EDuration="1.824209597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:45.819827268 +0000 UTC m=+1933.788646834" watchObservedRunningTime="2026-10-30 22:26:45.824209597 +0000 UTC m=+1933.793029187" Oct 30 22:26:45.956984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:45.956868 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" podStartSLOduration=1.956851626 podStartE2EDuration="1.956851626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:45.941961404 +0000 UTC m=+1933.910780996" watchObservedRunningTime="2026-10-30 22:26:45.956851626 +0000 UTC m=+1933.925671202" Oct 30 22:26:46.004553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:46.004513 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8"] Oct 30 22:26:46.008467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:46.008439 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-367f4491-f5bl8"] Oct 30 22:26:46.150119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:46.150082 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac05f11-9794-457b-9a35-efa4c2ff61b2" path="/var/lib/kubelet/pods/dac05f11-9794-457b-9a35-efa4c2ff61b2/volumes" Oct 30 22:26:54.262642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.262608 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:26:54.263257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.262811 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" containerID="cri-o://4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612" gracePeriod=30 Oct 30 22:26:54.276851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.276825 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:26:54.277059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.277033 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" containerID="cri-o://a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3" gracePeriod=30 Oct 30 22:26:54.303030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.303006 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:26:54.772707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:54.772669 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" containerID="cri-o://f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b" gracePeriod=30 Oct 30 22:26:56.562657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.562620 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:26:56.567905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.567886 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.622561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.622530 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-56c628-openshift-default-dockercfg-znr25\"" Oct 30 22:26:56.656488 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:56.656451 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:26:56.677828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.677799 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:26:56.688875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.688846 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.688954 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.688984 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689078 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689199 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689301 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klp9d\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689469 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689500 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.689561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.689534 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.709453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.709422 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:26:56.809469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809411 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809506 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809554 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809627 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809739 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809792 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.809913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809878 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.810066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.809977 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.810121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.810085 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klp9d\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.811649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.811618 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.812988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.812920 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.812988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.812970 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.813178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.812991 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.813350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.813328 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.815764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.815740 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.816130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.816108 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.882689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.882657 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klp9d\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:56.890201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:56.890144 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\") pod \"e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:57.183478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.183397 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:57.624280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.624245 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:26:57.657286 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:26:57.657239 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce900df_26aa_4630_8bca_bb946483a843.slice/crio-331c35c4e76e423f206da97e819bfef4f3242ea738609b69f33b87ada0923d7f WatchSource:0}: Error finding container 331c35c4e76e423f206da97e819bfef4f3242ea738609b69f33b87ada0923d7f: Status 404 returned error can't find the container with id 331c35c4e76e423f206da97e819bfef4f3242ea738609b69f33b87ada0923d7f Oct 30 22:26:57.659917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.659885 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:57.660000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.659955 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:57.660000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.659985 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:26:57.716061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.716038 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:26:57.808667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.808620 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerStarted","Data":"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389"} Oct 30 22:26:57.808835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.808688 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerStarted","Data":"331c35c4e76e423f206da97e819bfef4f3242ea738609b69f33b87ada0923d7f"} Oct 30 22:26:57.965561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:57.965452 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podStartSLOduration=1.9654320840000001 podStartE2EDuration="1.965432084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:57.958386152 +0000 UTC m=+1945.927205731" watchObservedRunningTime="2026-10-30 22:26:57.965432084 +0000 UTC m=+1945.934251650" Oct 30 22:26:58.183655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:58.183619 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:58.189954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:58.189927 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:58.816569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:58.816531 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:58.817519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:58.817500 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:26:59.581123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.581099 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:26:59.612768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.612747 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:26:59.650740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650715 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.650840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650789 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.650887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650851 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.650926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650896 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.650968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650948 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.650977 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651043 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651121 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651127 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651187 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651226 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651275 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651304 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651316 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651364 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651422 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651485 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651515 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651320 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651499 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651533 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651515 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651623 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.651666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651661 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\") pod \"28bef028-011b-4770-a463-b56c93bdc705\" (UID: \"28bef028-011b-4770-a463-b56c93bdc705\") " Oct 30 22:26:59.652087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651739 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\") pod \"c37ea491-7249-4452-93c8-d9bd30cddc76\" (UID: \"c37ea491-7249-4452-93c8-d9bd30cddc76\") " Oct 30 22:26:59.652087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651769 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.652087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.651784 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.652261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652090 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:59.652615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652567 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652622 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652640 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652654 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ea491-7249-4452-93c8-d9bd30cddc76-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652667 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652683 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.652718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652709 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.653056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652726 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.653056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652741 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bef028-011b-4770-a463-b56c93bdc705-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.653056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.652757 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.653592 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.653551 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:59.653719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.653692 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.653802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.653761 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.654179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.654157 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.654267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.654162 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "kube-api-access-gls8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.654450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.654426 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "kube-api-access-pnd8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.654881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.654856 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy" pod "c37ea491-7249-4452-93c8-d9bd30cddc76" (UID: "c37ea491-7249-4452-93c8-d9bd30cddc76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.654981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.654937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo" pod "28bef028-011b-4770-a463-b56c93bdc705" (UID: "28bef028-011b-4770-a463-b56c93bdc705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:59.754600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754529 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754560 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gls8m\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-kube-api-access-gls8m\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754588 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bef028-011b-4770-a463-b56c93bdc705-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754603 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bef028-011b-4770-a463-b56c93bdc705-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754618 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754631 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnd8w\" (UniqueName: \"kubernetes.io/projected/c37ea491-7249-4452-93c8-d9bd30cddc76-kube-api-access-pnd8w\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754644 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bef028-011b-4770-a463-b56c93bdc705-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.754893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.754657 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ea491-7249-4452-93c8-d9bd30cddc76-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.826619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.826587 4292 generic.go:356] "Generic (PLEG): container finished" podID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerID="4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612" exitCode=0 Oct 30 22:26:59.827098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.826635 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" event={"ID":"c37ea491-7249-4452-93c8-d9bd30cddc76","Type":"ContainerDied","Data":"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612"} Oct 30 22:26:59.827098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.826660 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" Oct 30 22:26:59.827098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.826702 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx" event={"ID":"c37ea491-7249-4452-93c8-d9bd30cddc76","Type":"ContainerDied","Data":"3f988acc5d53f108f4306525383460f2b42d2853094a4a2a29a9563322a65bee"} Oct 30 22:26:59.827098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.826732 4292 scope.go:122] "RemoveContainer" containerID="4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612" Oct 30 22:26:59.832613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.828319 4292 generic.go:356] "Generic (PLEG): container finished" podID="28bef028-011b-4770-a463-b56c93bdc705" containerID="a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3" exitCode=0 Oct 30 22:26:59.832613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.828363 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" Oct 30 22:26:59.832613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.828409 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" event={"ID":"28bef028-011b-4770-a463-b56c93bdc705","Type":"ContainerDied","Data":"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3"} Oct 30 22:26:59.832613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.828456 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7" event={"ID":"28bef028-011b-4770-a463-b56c93bdc705","Type":"ContainerDied","Data":"e36656c7de2ad26414ef0842ddc85d8d3f3119ca7269169a00c81106919648a3"} Oct 30 22:26:59.896084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.896054 4292 scope.go:122] "RemoveContainer" containerID="4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612" Oct 30 22:26:59.897851 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:59.897817 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612\": container with ID starting with 4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612 not found: ID does not exist" containerID="4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612" Oct 30 22:26:59.897972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.897866 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612"} err="failed to get container status \"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612\": rpc error: code = NotFound desc = could not find container \"4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612\": container with ID starting with 4ebcc20d81c490a7ee99cf8280c7697129f2c5011b08584fd1c1783cbd4b2612 not found: ID does not exist" Oct 30 22:26:59.897972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.897895 4292 scope.go:122] "RemoveContainer" containerID="a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3" Oct 30 22:26:59.914095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.914072 4292 scope.go:122] "RemoveContainer" containerID="a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3" Oct 30 22:26:59.914490 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:26:59.914460 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3\": container with ID starting with a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3 not found: ID does not exist" containerID="a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3" Oct 30 22:26:59.914634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:26:59.914503 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3"} err="failed to get container status \"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3\": rpc error: code = NotFound desc = could not find container \"a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3\": container with ID starting with a4a4fb8868e5db8a345311061ccc3ca815c827f75d1e98b4531b4d1d9d54bab3 not found: ID does not exist" Oct 30 22:27:00.008408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.008367 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:27:00.015163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.015126 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-lpqsx"] Oct 30 22:27:00.080096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.080066 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:27:00.122416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.122387 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:27:00.128144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.128116 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-nx2j7"] Oct 30 22:27:00.148999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.148968 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bef028-011b-4770-a463-b56c93bdc705" path="/var/lib/kubelet/pods/28bef028-011b-4770-a463-b56c93bdc705/volumes" Oct 30 22:27:00.149529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.149514 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" path="/var/lib/kubelet/pods/c37ea491-7249-4452-93c8-d9bd30cddc76/volumes" Oct 30 22:27:00.159906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.159886 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160048 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160094 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160196 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160249 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160322 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160356 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160369 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:00.160499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160421 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\") pod \"905153d1-5e1a-4828-9974-84c54b150d64\" (UID: \"905153d1-5e1a-4828-9974-84c54b150d64\") " Oct 30 22:27:00.160499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160466 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:00.160770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160747 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:00.160848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.160801 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:00.161055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161031 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:00.161204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161187 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.161275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161210 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.161275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161225 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905153d1-5e1a-4828-9974-84c54b150d64-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.161275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161238 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.161275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.161253 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.162268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.162248 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "kube-api-access-q9rrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:00.162712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.162691 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:00.162771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.162720 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:00.163128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.163111 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy" pod "905153d1-5e1a-4828-9974-84c54b150d64" (UID: "905153d1-5e1a-4828-9974-84c54b150d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:00.265621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.263313 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rrr\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-kube-api-access-q9rrr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.265621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.263352 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905153d1-5e1a-4828-9974-84c54b150d64-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.265621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.263368 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905153d1-5e1a-4828-9974-84c54b150d64-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.265621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.263381 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905153d1-5e1a-4828-9974-84c54b150d64-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.837665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.837625 4292 generic.go:356] "Generic (PLEG): container finished" podID="905153d1-5e1a-4828-9974-84c54b150d64" containerID="f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b" exitCode=0 Oct 30 22:27:00.838113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.837693 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" Oct 30 22:27:00.838113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.837711 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" event={"ID":"905153d1-5e1a-4828-9974-84c54b150d64","Type":"ContainerDied","Data":"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b"} Oct 30 22:27:00.838113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.837766 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl" event={"ID":"905153d1-5e1a-4828-9974-84c54b150d64","Type":"ContainerDied","Data":"50c4623e49a1dc0b9a3f852de543dc9262d245a29d3da4595b11b6391d71041c"} Oct 30 22:27:00.838113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.837796 4292 scope.go:122] "RemoveContainer" containerID="f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b" Oct 30 22:27:00.850126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.850103 4292 scope.go:122] "RemoveContainer" containerID="f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b" Oct 30 22:27:00.850403 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:27:00.850381 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b\": container with ID starting with f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b not found: ID does not exist" containerID="f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b" Oct 30 22:27:00.850480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:00.850413 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b"} err="failed to get container status \"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b\": rpc error: code = NotFound desc = could not find container \"f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b\": container with ID starting with f0a114d3276e06eb84746d47ff4af3ec4205dc6e5dce2e839147b66ba99ee67b not found: ID does not exist" Oct 30 22:27:01.019101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.019065 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:27:01.069042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.069013 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-367f4491-openshift-default-5f467bc7d9-gf2sl"] Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.354774 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356017 4292 cpu_manager.go:414] "containerMap: removing container" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356041 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356060 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356067 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356084 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356093 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356288 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="905153d1-5e1a-4828-9974-84c54b150d64" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356300 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37ea491-7249-4452-93c8-d9bd30cddc76" containerName="istio-proxy" Oct 30 22:27:01.356808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.356308 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bef028-011b-4770-a463-b56c93bdc705" containerName="istio-proxy" Oct 30 22:27:01.362600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.362001 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.382924 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2ps\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383082 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383147 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383186 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383243 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383297 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383341 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383371 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.392074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.383441 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.418343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.418250 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6d4d3c-openshift-default-dockercfg-m6phj\"" Oct 30 22:27:01.465626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.464676 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484631 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484687 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484714 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484750 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484775 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484801 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484820 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484860 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.484947 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2ps\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.485875 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.486035 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.486117 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.486266 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.488328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.486344 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.492815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.492788 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.493503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.493477 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.548488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.548456 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2ps\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.550457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.550431 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\") pod \"e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:01.744376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:01.744300 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:02.149320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.149288 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905153d1-5e1a-4828-9974-84c54b150d64" path="/var/lib/kubelet/pods/905153d1-5e1a-4828-9974-84c54b150d64/volumes" Oct 30 22:27:02.200120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.200089 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:27:02.228255 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:02.228217 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2754a886_2f9f_4515_9896_1fb4cbc58cb4.slice/crio-e1f08955878e50e5b40b68bdf0768d5f4dae83231587e679862258acd0c8a702 WatchSource:0}: Error finding container e1f08955878e50e5b40b68bdf0768d5f4dae83231587e679862258acd0c8a702: Status 404 returned error can't find the container with id e1f08955878e50e5b40b68bdf0768d5f4dae83231587e679862258acd0c8a702 Oct 30 22:27:02.231192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.231160 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:02.231272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.231232 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:02.231272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.231263 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:02.864623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.864531 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerStarted","Data":"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742"} Oct 30 22:27:02.872030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:02.871989 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerStarted","Data":"e1f08955878e50e5b40b68bdf0768d5f4dae83231587e679862258acd0c8a702"} Oct 30 22:27:03.007723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:03.007658 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podStartSLOduration=2.007639826 podStartE2EDuration="2.007639826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:03.006679866 +0000 UTC m=+1950.975499422" watchObservedRunningTime="2026-10-30 22:27:03.007639826 +0000 UTC m=+1950.976459392" Oct 30 22:27:03.745016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:03.744980 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:03.750570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:03.750545 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:03.867238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:03.867209 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:03.868215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:03.868185 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:27:09.646811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.645295 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:27:09.652544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.651553 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:09.689031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.688996 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:27:09.734099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.734061 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-56c628\"" Oct 30 22:27:09.800860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.800821 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-56c628-dockercfg-895tk\"" Oct 30 22:27:09.824622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.810295 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:27:09.873530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.827402 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:09.873530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.861228 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:27:09.876521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.876399 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92d9\" (UniqueName: \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:09.877860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.877816 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:09.979513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.979476 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:09.979746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.979561 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:09.979746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.979709 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvn8r\" (UniqueName: \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:09.979872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.979851 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92d9\" (UniqueName: \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:09.980266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:09.980247 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:10.043413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.043376 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92d9\" (UniqueName: \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\") pod \"payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:10.081413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.081381 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvn8r\" (UniqueName: \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.081621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.081596 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.082222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.082204 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.145392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.145361 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvn8r\" (UniqueName: \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\") pod \"payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.244994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.244918 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.267659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.267615 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:10.655173 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:10.655143 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe72a21d_df2c_4cbe_90e4_cfc491e7788c.slice/crio-9aa89558e3bc57e064e48707e03811480cb878823143a122053e2264edf20ef7 WatchSource:0}: Error finding container 9aa89558e3bc57e064e48707e03811480cb878823143a122053e2264edf20ef7: Status 404 returned error can't find the container with id 9aa89558e3bc57e064e48707e03811480cb878823143a122053e2264edf20ef7 Oct 30 22:27:10.655514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.655490 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:27:10.704264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.703247 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:27:10.721134 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:10.721107 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode217501d_0452_4108_8821_5ecb214095ee.slice/crio-fb01bbc277b692cc8674d4fe96c6fc742e64e75e7303a241f7e2073afe269d72 WatchSource:0}: Error finding container fb01bbc277b692cc8674d4fe96c6fc742e64e75e7303a241f7e2073afe269d72: Status 404 returned error can't find the container with id fb01bbc277b692cc8674d4fe96c6fc742e64e75e7303a241f7e2073afe269d72 Oct 30 22:27:10.932892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.932800 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" event={"ID":"be72a21d-df2c-4cbe-90e4-cfc491e7788c","Type":"ContainerStarted","Data":"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775"} Oct 30 22:27:10.932892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.932889 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:10.933110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.932912 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" event={"ID":"be72a21d-df2c-4cbe-90e4-cfc491e7788c","Type":"ContainerStarted","Data":"9aa89558e3bc57e064e48707e03811480cb878823143a122053e2264edf20ef7"} Oct 30 22:27:10.934518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.934490 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" event={"ID":"e217501d-0452-4108-8821-5ecb214095ee","Type":"ContainerStarted","Data":"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76"} Oct 30 22:27:10.934669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.934535 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" event={"ID":"e217501d-0452-4108-8821-5ecb214095ee","Type":"ContainerStarted","Data":"fb01bbc277b692cc8674d4fe96c6fc742e64e75e7303a241f7e2073afe269d72"} Oct 30 22:27:10.934669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:10.934558 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:11.056602 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:11.056514 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" podStartSLOduration=2.056496736 podStartE2EDuration="2.056496736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.052209025 +0000 UTC m=+1959.021028589" watchObservedRunningTime="2026-10-30 22:27:11.056496736 +0000 UTC m=+1959.025316302" Oct 30 22:27:11.173597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:11.172291 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" podStartSLOduration=2.172273296 podStartE2EDuration="2.172273296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.171433491 +0000 UTC m=+1959.140253056" watchObservedRunningTime="2026-10-30 22:27:11.172273296 +0000 UTC m=+1959.141092863" Oct 30 22:27:12.231197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.231162 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:27:12.235879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.235852 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.293462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.293432 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:27:12.321294 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.321258 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9ns\" (UniqueName: \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.322386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.321372 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.343162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.339733 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6d4d3c\"" Oct 30 22:27:12.343162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.339771 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6d4d3c-dockercfg-kqqm5\"" Oct 30 22:27:12.412621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.411912 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:27:12.417960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.417926 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.423334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.423276 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9ns\" (UniqueName: \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.423482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.423396 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.424448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.424424 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.443893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.443871 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:27:12.487512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.487438 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9ns\" (UniqueName: \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\") pod \"payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.531831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.528698 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrnh\" (UniqueName: \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.531831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.528828 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.571596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.554871 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:12.632114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.630688 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrnh\" (UniqueName: \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.632114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.630812 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.632114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.631787 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.695750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.695721 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrnh\" (UniqueName: \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\") pod \"payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:12.734401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:12.734367 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:13.002080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:13.002049 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:27:13.002393 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:13.002369 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901c94fa_a358_4834_80a9_7a980f08703c.slice/crio-c595294687ca3bd3105e019e98377da8a9b93558297f681a30f9cd12d12a09c4 WatchSource:0}: Error finding container c595294687ca3bd3105e019e98377da8a9b93558297f681a30f9cd12d12a09c4: Status 404 returned error can't find the container with id c595294687ca3bd3105e019e98377da8a9b93558297f681a30f9cd12d12a09c4 Oct 30 22:27:13.024803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:13.024770 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" event={"ID":"901c94fa-a358-4834-80a9-7a980f08703c","Type":"ContainerStarted","Data":"c595294687ca3bd3105e019e98377da8a9b93558297f681a30f9cd12d12a09c4"} Oct 30 22:27:13.149966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:13.149930 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:27:13.157633 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:13.157606 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9903bac3_efc9_4de3_b4ea_e84a7fc853da.slice/crio-525291b5555bc534694274469afc9705ce274aa282df31176288a7cdd41741ad WatchSource:0}: Error finding container 525291b5555bc534694274469afc9705ce274aa282df31176288a7cdd41741ad: Status 404 returned error can't find the container with id 525291b5555bc534694274469afc9705ce274aa282df31176288a7cdd41741ad Oct 30 22:27:14.034411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.034368 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" event={"ID":"901c94fa-a358-4834-80a9-7a980f08703c","Type":"ContainerStarted","Data":"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43"} Oct 30 22:27:14.034892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.034439 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:14.036197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.036167 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" event={"ID":"9903bac3-efc9-4de3-b4ea-e84a7fc853da","Type":"ContainerStarted","Data":"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b"} Oct 30 22:27:14.036320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.036215 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" event={"ID":"9903bac3-efc9-4de3-b4ea-e84a7fc853da","Type":"ContainerStarted","Data":"525291b5555bc534694274469afc9705ce274aa282df31176288a7cdd41741ad"} Oct 30 22:27:14.036320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.036240 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:14.156722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.156667 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" podStartSLOduration=2.156654315 podStartE2EDuration="2.156654315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:14.1540116 +0000 UTC m=+1962.122831177" watchObservedRunningTime="2026-10-30 22:27:14.156654315 +0000 UTC m=+1962.125473878" Oct 30 22:27:14.290424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:14.290292 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" podStartSLOduration=2.29027141 podStartE2EDuration="2.29027141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:14.282037556 +0000 UTC m=+1962.250857133" watchObservedRunningTime="2026-10-30 22:27:14.29027141 +0000 UTC m=+1962.259090975" Oct 30 22:27:22.035154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:22.034961 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:27:22.047824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:22.047794 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:27:23.580524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:23.580492 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:27:23.580934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:23.580695 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" containerID="cri-o://39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278" gracePeriod=30 Oct 30 22:27:24.301127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.301082 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5"] Oct 30 22:27:24.308716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.308686 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.366279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.366242 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:24.366453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.366397 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:27:24.368034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.366463 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:24.368034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.366552 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:24.372999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.372968 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5"] Oct 30 22:27:24.455886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.455855 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.456058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.455968 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ww84\" (UniqueName: \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.456058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.456018 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.557875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.557784 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.558035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.557902 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ww84\" (UniqueName: \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.558035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.557950 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.558664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.558636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.560281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.560260 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.624327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.624293 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ww84\" (UniqueName: \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:24.625077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:24.625056 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:25.006782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:25.006746 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5"] Oct 30 22:27:25.006956 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:25.006913 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc89326_3fc6_4379_bf75_c6fac878fee5.slice/crio-1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01 WatchSource:0}: Error finding container 1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01: Status 404 returned error can't find the container with id 1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01 Oct 30 22:27:25.046775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:25.046753 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:27:25.046886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:25.046826 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:27:25.158733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:25.158703 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" event={"ID":"5cc89326-3fc6-4379-bf75-c6fac878fee5","Type":"ContainerStarted","Data":"1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01"} Oct 30 22:27:26.167362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:26.167324 4292 generic.go:356] "Generic (PLEG): container finished" podID="5cc89326-3fc6-4379-bf75-c6fac878fee5" containerID="f8917b8a39b615b7e01d94afa7ccb46e70e1dabf1c34a8052ab2005dccee82f2" exitCode=0 Oct 30 22:27:26.167808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:26.167428 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" event={"ID":"5cc89326-3fc6-4379-bf75-c6fac878fee5","Type":"ContainerDied","Data":"f8917b8a39b615b7e01d94afa7ccb46e70e1dabf1c34a8052ab2005dccee82f2"} Oct 30 22:27:26.641883 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:27:26.641849 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:27:27.350600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.350558 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:27.402013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.401987 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\") pod \"5cc89326-3fc6-4379-bf75-c6fac878fee5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " Oct 30 22:27:27.402159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.402084 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\") pod \"5cc89326-3fc6-4379-bf75-c6fac878fee5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " Oct 30 22:27:27.402159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.402109 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\" (UniqueName: \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\") pod \"5cc89326-3fc6-4379-bf75-c6fac878fee5\" (UID: \"5cc89326-3fc6-4379-bf75-c6fac878fee5\") " Oct 30 22:27:27.402484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.402459 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca" pod "5cc89326-3fc6-4379-bf75-c6fac878fee5" (UID: "5cc89326-3fc6-4379-bf75-c6fac878fee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:27.404242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.404219 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca" pod "5cc89326-3fc6-4379-bf75-c6fac878fee5" (UID: "5cc89326-3fc6-4379-bf75-c6fac878fee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:27.404337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.404233 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84" pod "5cc89326-3fc6-4379-bf75-c6fac878fee5" (UID: "5cc89326-3fc6-4379-bf75-c6fac878fee5"). InnerVolumeSpecName "kube-api-access-2ww84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:27.503476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.503398 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:27.503476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.503426 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc89326-3fc6-4379-bf75-c6fac878fee5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:27.503476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:27.503439 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ww84\" (UniqueName: \"kubernetes.io/projected/5cc89326-3fc6-4379-bf75-c6fac878fee5-kube-api-access-2ww84\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.185033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.185008 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" Oct 30 22:27:28.185202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.185080 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5" event={"ID":"5cc89326-3fc6-4379-bf75-c6fac878fee5","Type":"ContainerDied","Data":"1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01"} Oct 30 22:27:28.185202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.185125 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5b02382a0a1f22d9a90f42484b8ba237e4079a1cb3802b9adb899859bbef01" Oct 30 22:27:28.869831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.869805 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:27:28.923281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923314 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923370 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923394 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923423 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923520 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923553 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.923812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923643 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.923812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923740 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.924016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923836 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.924016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923852 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.924016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.923930 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:28.924174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924070 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\") pod \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\" (UID: \"d35b8b33-0e4d-43ee-8242-fd41159c9dab\") " Oct 30 22:27:28.924780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924752 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.924899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924783 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.924899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924800 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.924899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924814 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.924899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.924829 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.925651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.925620 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "kube-api-access-fghx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.925781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.925765 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.925879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.925855 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:28.926085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:28.926072 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy" pod "d35b8b33-0e4d-43ee-8242-fd41159c9dab" (UID: "d35b8b33-0e4d-43ee-8242-fd41159c9dab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:29.026626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.026564 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:29.026626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.026625 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:29.026796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.026641 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghx9\" (UniqueName: \"kubernetes.io/projected/d35b8b33-0e4d-43ee-8242-fd41159c9dab-kube-api-access-fghx9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:29.026796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.026655 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b8b33-0e4d-43ee-8242-fd41159c9dab-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:29.194713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.194626 4292 generic.go:356] "Generic (PLEG): container finished" podID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerID="39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278" exitCode=0 Oct 30 22:27:29.194713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.194678 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" Oct 30 22:27:29.194931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.194710 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" event={"ID":"d35b8b33-0e4d-43ee-8242-fd41159c9dab","Type":"ContainerDied","Data":"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278"} Oct 30 22:27:29.194931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.194761 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc" event={"ID":"d35b8b33-0e4d-43ee-8242-fd41159c9dab","Type":"ContainerDied","Data":"6ab132fcb0eadd4b87933aa583eb821c2b6450c522db8f95b2d571e9322ebb81"} Oct 30 22:27:29.194931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.194782 4292 scope.go:122] "RemoveContainer" containerID="39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278" Oct 30 22:27:29.207633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.207616 4292 scope.go:122] "RemoveContainer" containerID="39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278" Oct 30 22:27:29.207911 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:27:29.207892 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278\": container with ID starting with 39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278 not found: ID does not exist" containerID="39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278" Oct 30 22:27:29.207959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.207920 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278"} err="failed to get container status \"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278\": rpc error: code = NotFound desc = could not find container \"39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278\": container with ID starting with 39251ca16ed73feba873fa3924f3ed06994218a5beffdd87a5072f1715b0a278 not found: ID does not exist" Oct 30 22:27:29.372250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.372217 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:27:29.383463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:29.383433 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-drqkc"] Oct 30 22:27:30.150557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:30.150518 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" path="/var/lib/kubelet/pods/d35b8b33-0e4d-43ee-8242-fd41159c9dab/volumes" Oct 30 22:27:50.003254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.003220 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk"] Oct 30 22:27:50.004032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004013 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5cc89326-3fc6-4379-bf75-c6fac878fee5" containerName="revoke-keys" Oct 30 22:27:50.004085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004033 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc89326-3fc6-4379-bf75-c6fac878fee5" containerName="revoke-keys" Oct 30 22:27:50.004085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004077 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" Oct 30 22:27:50.004085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004082 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" Oct 30 22:27:50.004212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004202 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc89326-3fc6-4379-bf75-c6fac878fee5" containerName="revoke-keys" Oct 30 22:27:50.004247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.004220 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35b8b33-0e4d-43ee-8242-fd41159c9dab" containerName="istio-proxy" Oct 30 22:27:50.009641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.009621 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.068407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.068378 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk"] Oct 30 22:27:50.112879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.112853 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:50.113016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.112896 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:50.113934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.113917 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:50.114031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.114010 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:27:50.129030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.129004 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.129129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.129110 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8q28\" (UniqueName: \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.129410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.129389 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.231393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.231346 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.231549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.231467 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.231549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.231504 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q28\" (UniqueName: \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.232025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.232007 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.233773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.233757 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.294272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.294219 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8q28\" (UniqueName: \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\") pod \"maas-api-revoke-keys-e2e-api-a-a42692-msblk\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.324920 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.324900 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:50.706864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:50.706833 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk"] Oct 30 22:27:50.707479 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:27:50.707454 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0a5a59_22ca_4089_a2fc_591b018f6bd9.slice/crio-0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe WatchSource:0}: Error finding container 0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe: Status 404 returned error can't find the container with id 0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe Oct 30 22:27:51.380067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:51.380030 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" event={"ID":"fa0a5a59-22ca-4089-a2fc-591b018f6bd9","Type":"ContainerStarted","Data":"33a6bdca3f74a37aa5b962126528aeb3993f28d8c0ef5cef5116d49f87a87da6"} Oct 30 22:27:51.380461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:51.380090 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" event={"ID":"fa0a5a59-22ca-4089-a2fc-591b018f6bd9","Type":"ContainerStarted","Data":"0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe"} Oct 30 22:27:51.502888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:51.502534 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" podStartSLOduration=2.502516956 podStartE2EDuration="2.502516956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:51.498436255 +0000 UTC m=+1999.467255819" watchObservedRunningTime="2026-10-30 22:27:51.502516956 +0000 UTC m=+1999.471336521" Oct 30 22:27:52.388583 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:52.388548 4292 generic.go:356] "Generic (PLEG): container finished" podID="fa0a5a59-22ca-4089-a2fc-591b018f6bd9" containerID="33a6bdca3f74a37aa5b962126528aeb3993f28d8c0ef5cef5116d49f87a87da6" exitCode=0 Oct 30 22:27:52.389013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:52.388643 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" event={"ID":"fa0a5a59-22ca-4089-a2fc-591b018f6bd9","Type":"ContainerDied","Data":"33a6bdca3f74a37aa5b962126528aeb3993f28d8c0ef5cef5116d49f87a87da6"} Oct 30 22:27:53.581959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.581933 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:53.677700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.677628 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\" (UniqueName: \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\") pod \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " Oct 30 22:27:53.677836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.677762 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\") pod \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " Oct 30 22:27:53.677836 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.677826 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\") pod \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\" (UID: \"fa0a5a59-22ca-4089-a2fc-591b018f6bd9\") " Oct 30 22:27:53.678156 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.678127 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca" pod "fa0a5a59-22ca-4089-a2fc-591b018f6bd9" (UID: "fa0a5a59-22ca-4089-a2fc-591b018f6bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:53.678508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.678486 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:53.679911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.679886 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28" pod "fa0a5a59-22ca-4089-a2fc-591b018f6bd9" (UID: "fa0a5a59-22ca-4089-a2fc-591b018f6bd9"). InnerVolumeSpecName "kube-api-access-t8q28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:53.680006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.679953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca" pod "fa0a5a59-22ca-4089-a2fc-591b018f6bd9" (UID: "fa0a5a59-22ca-4089-a2fc-591b018f6bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:53.779913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.779883 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:53.779913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:53.779910 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8q28\" (UniqueName: \"kubernetes.io/projected/fa0a5a59-22ca-4089-a2fc-591b018f6bd9-kube-api-access-t8q28\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:54.405118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:54.405088 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" Oct 30 22:27:54.405118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:54.405109 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk" event={"ID":"fa0a5a59-22ca-4089-a2fc-591b018f6bd9","Type":"ContainerDied","Data":"0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe"} Oct 30 22:27:54.405317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:27:54.405152 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e383fcef5f44633fad5f5cc6703ba401dc6b801e142c87d429d1a9ccece76fe" Oct 30 22:27:56.640425 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:27:56.640388 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:28:03.973665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:03.973631 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:28:04.088136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:04.088101 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:28:08.063542 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:08.063493 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc"] Oct 30 22:28:08.112268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:08.112236 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0140d87b-7q2nc"] Oct 30 22:28:08.148853 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:08.148822 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328" path="/var/lib/kubelet/pods/3ec8624e-d4fb-46d7-9a2a-a5ddc15f8328/volumes" Oct 30 22:28:08.974486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:08.974436 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" podUID="3d9924d8-7c90-458a-933b-98037875436d" containerName="payload-processing" containerID="cri-o://98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73" gracePeriod=25 Oct 30 22:28:09.088751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.088705 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" podUID="da8b4344-68fd-41b5-b339-6bb048428101" containerName="payload-pre-processing" containerID="cri-o://4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9" gracePeriod=25 Oct 30 22:28:09.264988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.264958 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:28:09.302058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.301521 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\") pod \"3d9924d8-7c90-458a-933b-98037875436d\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " Oct 30 22:28:09.302058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.301761 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\" (UniqueName: \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\") pod \"3d9924d8-7c90-458a-933b-98037875436d\" (UID: \"3d9924d8-7c90-458a-933b-98037875436d\") " Oct 30 22:28:09.305002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.303417 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume" pod "3d9924d8-7c90-458a-933b-98037875436d" (UID: "3d9924d8-7c90-458a-933b-98037875436d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:09.312896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.312863 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws" pod "3d9924d8-7c90-458a-933b-98037875436d" (UID: "3d9924d8-7c90-458a-933b-98037875436d"). InnerVolumeSpecName "kube-api-access-x7wws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:09.395095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.395066 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:28:09.403857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.403825 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wws\" (UniqueName: \"kubernetes.io/projected/3d9924d8-7c90-458a-933b-98037875436d-kube-api-access-x7wws\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.403984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.403860 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9924d8-7c90-458a-933b-98037875436d-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.504436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.504360 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\" (UniqueName: \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\") pod \"da8b4344-68fd-41b5-b339-6bb048428101\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " Oct 30 22:28:09.504436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.504414 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\") pod \"da8b4344-68fd-41b5-b339-6bb048428101\" (UID: \"da8b4344-68fd-41b5-b339-6bb048428101\") " Oct 30 22:28:09.504797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.504773 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume" pod "da8b4344-68fd-41b5-b339-6bb048428101" (UID: "da8b4344-68fd-41b5-b339-6bb048428101"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:09.505150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.505124 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b4344-68fd-41b5-b339-6bb048428101-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.506455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.506429 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx" pod "da8b4344-68fd-41b5-b339-6bb048428101" (UID: "da8b4344-68fd-41b5-b339-6bb048428101"). InnerVolumeSpecName "kube-api-access-72kxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:09.581647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.581609 4292 generic.go:356] "Generic (PLEG): container finished" podID="da8b4344-68fd-41b5-b339-6bb048428101" containerID="4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9" exitCode=0 Oct 30 22:28:09.581911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.581676 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" Oct 30 22:28:09.582046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.581693 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" event={"ID":"da8b4344-68fd-41b5-b339-6bb048428101","Type":"ContainerDied","Data":"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9"} Oct 30 22:28:09.582200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.582179 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj" event={"ID":"da8b4344-68fd-41b5-b339-6bb048428101","Type":"ContainerDied","Data":"ec5f8f42036f301c7082e4ceeb8a8b7e48a5743d9b9d1335bcb8bd04bdf03a59"} Oct 30 22:28:09.582315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.582302 4292 scope.go:122] "RemoveContainer" containerID="4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9" Oct 30 22:28:09.584318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.584271 4292 generic.go:356] "Generic (PLEG): container finished" podID="3d9924d8-7c90-458a-933b-98037875436d" containerID="98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73" exitCode=0 Oct 30 22:28:09.584402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.584346 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" event={"ID":"3d9924d8-7c90-458a-933b-98037875436d","Type":"ContainerDied","Data":"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73"} Oct 30 22:28:09.584402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.584386 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" event={"ID":"3d9924d8-7c90-458a-933b-98037875436d","Type":"ContainerDied","Data":"c828c6c6f5937c2ffd212482930c5caef56079c1f31eded3397a6b11b8fd5cec"} Oct 30 22:28:09.584519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.584487 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7" Oct 30 22:28:09.596671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.596645 4292 scope.go:122] "RemoveContainer" containerID="4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9" Oct 30 22:28:09.597012 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:09.596991 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9\": container with ID starting with 4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9 not found: ID does not exist" containerID="4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9" Oct 30 22:28:09.597091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.597018 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9"} err="failed to get container status \"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9\": rpc error: code = NotFound desc = could not find container \"4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9\": container with ID starting with 4377ca96d037ade67144269253cf9407057b34402049232640820117321a21c9 not found: ID does not exist" Oct 30 22:28:09.597091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.597035 4292 scope.go:122] "RemoveContainer" containerID="98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73" Oct 30 22:28:09.606685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.606665 4292 scope.go:122] "RemoveContainer" containerID="98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73" Oct 30 22:28:09.606981 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:09.606960 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73\": container with ID starting with 98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73 not found: ID does not exist" containerID="98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73" Oct 30 22:28:09.607055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.606987 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73"} err="failed to get container status \"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73\": rpc error: code = NotFound desc = could not find container \"98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73\": container with ID starting with 98ce7f2c816d7220d823ae3a23c44719fa609bc91e74a85914c2eaa459503e73 not found: ID does not exist" Oct 30 22:28:09.607128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.607108 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72kxx\" (UniqueName: \"kubernetes.io/projected/da8b4344-68fd-41b5-b339-6bb048428101-kube-api-access-72kxx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.761817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.761728 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:28:09.765736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.765707 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a42692-5c6b4975cf-6f6m7"] Oct 30 22:28:09.882968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.882924 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:28:09.892295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:09.892259 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a42692-6bb8b8ffcf-4nzgj"] Oct 30 22:28:10.150376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:10.150342 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9924d8-7c90-458a-933b-98037875436d" path="/var/lib/kubelet/pods/3d9924d8-7c90-458a-933b-98037875436d/volumes" Oct 30 22:28:10.151025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:10.151001 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8b4344-68fd-41b5-b339-6bb048428101" path="/var/lib/kubelet/pods/da8b4344-68fd-41b5-b339-6bb048428101/volumes" Oct 30 22:28:16.519772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.519695 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt"] Oct 30 22:28:16.520384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520370 4292 cpu_manager.go:414] "containerMap: removing container" podUID="da8b4344-68fd-41b5-b339-6bb048428101" containerName="payload-pre-processing" Oct 30 22:28:16.520434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520385 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b4344-68fd-41b5-b339-6bb048428101" containerName="payload-pre-processing" Oct 30 22:28:16.520434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520420 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3d9924d8-7c90-458a-933b-98037875436d" containerName="payload-processing" Oct 30 22:28:16.520434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520426 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9924d8-7c90-458a-933b-98037875436d" containerName="payload-processing" Oct 30 22:28:16.520528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520447 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fa0a5a59-22ca-4089-a2fc-591b018f6bd9" containerName="revoke-keys" Oct 30 22:28:16.520528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520452 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0a5a59-22ca-4089-a2fc-591b018f6bd9" containerName="revoke-keys" Oct 30 22:28:16.520632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520590 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9924d8-7c90-458a-933b-98037875436d" containerName="payload-processing" Oct 30 22:28:16.520632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520605 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0a5a59-22ca-4089-a2fc-591b018f6bd9" containerName="revoke-keys" Oct 30 22:28:16.520632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.520615 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b4344-68fd-41b5-b339-6bb048428101" containerName="payload-pre-processing" Oct 30 22:28:16.525413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.525394 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.582822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.582797 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt"] Oct 30 22:28:16.604532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.604504 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.604689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.604666 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.604751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.604707 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrdz\" (UniqueName: \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.628927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.628891 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:16.629046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.629014 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:16.629122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.629063 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:28:16.629453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.629436 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:16.706680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.706646 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.706804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.706762 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.706904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.706813 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrdz\" (UniqueName: \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.707414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.707389 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.709434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.709410 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.770027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.769974 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrdz\" (UniqueName: \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\") pod \"maas-api-revoke-keys-e2e-mt-18515933-fz2qt\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:16.843449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:16.843428 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:17.225210 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:28:17.225179 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea0748a_761d_43f2_a86b_42ac45a55146.slice/crio-7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9 WatchSource:0}: Error finding container 7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9: Status 404 returned error can't find the container with id 7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9 Oct 30 22:28:17.225354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:17.225338 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt"] Oct 30 22:28:17.649135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:17.649097 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" event={"ID":"6ea0748a-761d-43f2-a86b-42ac45a55146","Type":"ContainerStarted","Data":"00061fbd25a0d7b5f29a0ef98f954c02db5984c8571c420d17b7eda0ec14679f"} Oct 30 22:28:17.649645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:17.649161 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" event={"ID":"6ea0748a-761d-43f2-a86b-42ac45a55146","Type":"ContainerStarted","Data":"7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9"} Oct 30 22:28:17.796668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:17.796615 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" podStartSLOduration=1.796597737 podStartE2EDuration="1.796597737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:17.791763053 +0000 UTC m=+2025.760582620" watchObservedRunningTime="2026-10-30 22:28:17.796597737 +0000 UTC m=+2025.765417303" Oct 30 22:28:18.659278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:18.659242 4292 generic.go:356] "Generic (PLEG): container finished" podID="6ea0748a-761d-43f2-a86b-42ac45a55146" containerID="00061fbd25a0d7b5f29a0ef98f954c02db5984c8571c420d17b7eda0ec14679f" exitCode=0 Oct 30 22:28:18.659746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:18.659323 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" event={"ID":"6ea0748a-761d-43f2-a86b-42ac45a55146","Type":"ContainerDied","Data":"00061fbd25a0d7b5f29a0ef98f954c02db5984c8571c420d17b7eda0ec14679f"} Oct 30 22:28:19.841474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.841453 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:19.951597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.951493 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\") pod \"6ea0748a-761d-43f2-a86b-42ac45a55146\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " Oct 30 22:28:19.951791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.951776 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\" (UniqueName: \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\") pod \"6ea0748a-761d-43f2-a86b-42ac45a55146\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " Oct 30 22:28:19.951847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.951829 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\") pod \"6ea0748a-761d-43f2-a86b-42ac45a55146\" (UID: \"6ea0748a-761d-43f2-a86b-42ac45a55146\") " Oct 30 22:28:19.952218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.952190 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca" pod "6ea0748a-761d-43f2-a86b-42ac45a55146" (UID: "6ea0748a-761d-43f2-a86b-42ac45a55146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:19.952440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.952423 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:19.953435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.953410 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca" pod "6ea0748a-761d-43f2-a86b-42ac45a55146" (UID: "6ea0748a-761d-43f2-a86b-42ac45a55146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:19.953675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:19.953659 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz" pod "6ea0748a-761d-43f2-a86b-42ac45a55146" (UID: "6ea0748a-761d-43f2-a86b-42ac45a55146"). InnerVolumeSpecName "kube-api-access-4wrdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:20.053710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:20.053682 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wrdz\" (UniqueName: \"kubernetes.io/projected/6ea0748a-761d-43f2-a86b-42ac45a55146-kube-api-access-4wrdz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.053710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:20.053708 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea0748a-761d-43f2-a86b-42ac45a55146-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.675933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:20.675897 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" Oct 30 22:28:20.675933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:20.675924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt" event={"ID":"6ea0748a-761d-43f2-a86b-42ac45a55146","Type":"ContainerDied","Data":"7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9"} Oct 30 22:28:20.676153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:20.675978 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f34ed4d189318b2d608f9719fbf872dda5ff269094d16ef2ccbeb233d7324d9" Oct 30 22:28:24.869626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:24.869588 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:28:24.870080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:24.869820 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" containerID="cri-o://933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72" gracePeriod=30 Oct 30 22:28:25.019566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:25.019534 4292 patch_prober.go:29] interesting pod/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:28:25.019735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:25.019612 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:28:25.156223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:25.156136 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:28:25.156744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:25.156708 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" containerID="cri-o://04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e" gracePeriod=30 Oct 30 22:28:26.640810 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:26.640769 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:28:26.647460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:26.647430 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:28:26.647620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:26.647496 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:28:27.669452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:27.669417 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:28:27.784174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:27.784140 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:28:30.150678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.150607 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:28:30.197931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.197901 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198028 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198061 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198104 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198234 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198247 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:30.198356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198301 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198334 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198339 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.198481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198364 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\") pod \"efc8ef88-62d4-42af-9726-09c6d854955c\" (UID: \"efc8ef88-62d4-42af-9726-09c6d854955c\") " Oct 30 22:28:30.198481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.198932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.198898 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.199129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.199105 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc8ef88-62d4-42af-9726-09c6d854955c-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.199205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.199126 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.199205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.199136 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.199205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.199173 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.199205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.199192 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.200559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.200533 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:30.201441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.200706 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "kube-api-access-xd5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:30.201441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.200817 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:30.201719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.201685 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy" pod "efc8ef88-62d4-42af-9726-09c6d854955c" (UID: "efc8ef88-62d4-42af-9726-09c6d854955c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.300532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.300501 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.300532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.300529 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.300738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.300539 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5q6\" (UniqueName: \"kubernetes.io/projected/efc8ef88-62d4-42af-9726-09c6d854955c-kube-api-access-xd5q6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.300738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.300550 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc8ef88-62d4-42af-9726-09c6d854955c-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.300738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.300563 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc8ef88-62d4-42af-9726-09c6d854955c-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.423568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.423544 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:28:30.503707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.503677 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.503901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.503727 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.503901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.503793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.503901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.503841 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.503927 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504009 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504053 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504154 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504201 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\") pod \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\" (UID: \"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b\") " Oct 30 22:28:30.504346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504217 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.504346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:30.504514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.504608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504535 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.504709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504687 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.505009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.504987 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.505053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.505018 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.505053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.505035 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.505116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.505050 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.505116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.505065 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.506122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.506098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:30.506261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.506244 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:30.506368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.506353 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "kube-api-access-44pb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:30.506439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.506422 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo" pod "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" (UID: "8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:30.606949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.606915 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.606949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.606947 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.607138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.606960 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pb7\" (UniqueName: \"kubernetes.io/projected/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-kube-api-access-44pb7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.607138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.606969 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.757491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.757408 4292 generic.go:356] "Generic (PLEG): container finished" podID="efc8ef88-62d4-42af-9726-09c6d854955c" containerID="933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72" exitCode=0 Oct 30 22:28:30.757491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.757457 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" Oct 30 22:28:30.757756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.757501 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" event={"ID":"efc8ef88-62d4-42af-9726-09c6d854955c","Type":"ContainerDied","Data":"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72"} Oct 30 22:28:30.757756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.757561 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz" event={"ID":"efc8ef88-62d4-42af-9726-09c6d854955c","Type":"ContainerDied","Data":"e671f6d89b968dc494d936e6b294edf79a5ae3400667450d0a31c32e3712a050"} Oct 30 22:28:30.757756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.757625 4292 scope.go:122] "RemoveContainer" containerID="933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72" Oct 30 22:28:30.759098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.759074 4292 generic.go:356] "Generic (PLEG): container finished" podID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerID="04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e" exitCode=0 Oct 30 22:28:30.759212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.759101 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" event={"ID":"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b","Type":"ContainerDied","Data":"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e"} Oct 30 22:28:30.759212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.759117 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" Oct 30 22:28:30.759212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.759153 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m" event={"ID":"8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b","Type":"ContainerDied","Data":"38a8489867151a91c5f4344f460f69c0bf9ae1ea80b5a69b3c6fd82fd7bd1603"} Oct 30 22:28:30.768603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.768569 4292 scope.go:122] "RemoveContainer" containerID="933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72" Oct 30 22:28:30.769010 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:30.768884 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72\": container with ID starting with 933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72 not found: ID does not exist" containerID="933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72" Oct 30 22:28:30.769010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.768906 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72"} err="failed to get container status \"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72\": rpc error: code = NotFound desc = could not find container \"933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72\": container with ID starting with 933d34fc9235500886b322729f0cc3c70f8374bda1ccac5d4d9a313ddc5f4a72 not found: ID does not exist" Oct 30 22:28:30.769010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.768922 4292 scope.go:122] "RemoveContainer" containerID="04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e" Oct 30 22:28:30.778923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.778907 4292 scope.go:122] "RemoveContainer" containerID="04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e" Oct 30 22:28:30.779187 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:30.779169 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e\": container with ID starting with 04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e not found: ID does not exist" containerID="04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e" Oct 30 22:28:30.779248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.779193 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e"} err="failed to get container status \"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e\": rpc error: code = NotFound desc = could not find container \"04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e\": container with ID starting with 04f6331ccd10b6869ccb410c49103e91cee169627ee632ce336fcb31e55a7a8e not found: ID does not exist" Oct 30 22:28:30.930884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.930852 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:28:30.981245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:30.981216 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-ld5nz"] Oct 30 22:28:31.095111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:31.095081 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:28:31.101538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:31.101516 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-8zf4m"] Oct 30 22:28:32.148595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:32.148547 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" path="/var/lib/kubelet/pods/8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b/volumes" Oct 30 22:28:32.149221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:32.149199 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" path="/var/lib/kubelet/pods/efc8ef88-62d4-42af-9726-09c6d854955c/volumes" Oct 30 22:28:32.670062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:32.670023 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" containerID="cri-o://95873856803e048c64e9c95fca879a785a8157a1a781798a9bd8e3faf9a694c3" gracePeriod=25 Oct 30 22:28:32.785245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:32.785205 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" containerID="cri-o://028a8f55a2896c67004897e95140bac9d7089525d10c5dd490122b784d25b78a" gracePeriod=25 Oct 30 22:28:33.081037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:33.080998 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:37.036169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:37.036132 4292 scope.go:122] "RemoveContainer" containerID="47d27a6fb80e98ffa1791758e6899c45e6917a43f1ac466f2f86fdd6af784064" Oct 30 22:28:40.763253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:40.763218 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5"] Oct 30 22:28:40.816083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:40.816042 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2dd57a-r5gz5"] Oct 30 22:28:42.053471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:42.053433 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:42.148715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:42.148678 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc89326-3fc6-4379-bf75-c6fac878fee5" path="/var/lib/kubelet/pods/5cc89326-3fc6-4379-bf75-c6fac878fee5/volumes" Oct 30 22:28:43.080924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:43.080883 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:51.765155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765118 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p"] Oct 30 22:28:51.765835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765820 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6ea0748a-761d-43f2-a86b-42ac45a55146" containerName="revoke-keys" Oct 30 22:28:51.765885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765836 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea0748a-761d-43f2-a86b-42ac45a55146" containerName="revoke-keys" Oct 30 22:28:51.765885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765883 4292 cpu_manager.go:414] "containerMap: removing container" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" Oct 30 22:28:51.765954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765889 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" Oct 30 22:28:51.765954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765897 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" Oct 30 22:28:51.765954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.765901 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" Oct 30 22:28:51.766047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.766005 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea0748a-761d-43f2-a86b-42ac45a55146" containerName="revoke-keys" Oct 30 22:28:51.766047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.766021 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc8ef88-62d4-42af-9726-09c6d854955c" containerName="istio-proxy" Oct 30 22:28:51.766047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.766027 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4e1a90-f3b3-4d16-a7fb-8ccd08d6b89b" containerName="istio-proxy" Oct 30 22:28:51.771390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.771372 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:51.828908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.828877 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p"] Oct 30 22:28:51.874272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.874241 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:51.874467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.874248 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:51.876129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.876107 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:51.876235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.876172 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:28:51.966491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.966458 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:51.966730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.966709 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:51.966783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:51.966741 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfwf\" (UniqueName: \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.054223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.054184 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:52.068019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.067985 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.068019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.068021 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfwf\" (UniqueName: \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.068201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.068108 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.068667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.068644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.070448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.070423 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.131267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.131232 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfwf\" (UniqueName: \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.409861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.409769 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:52.811326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.809670 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p"] Oct 30 22:28:52.850593 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:28:52.850527 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf862b947_3965_4ed3_bcfe_756bfcbb2487.slice/crio-03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b WatchSource:0}: Error finding container 03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b: Status 404 returned error can't find the container with id 03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b Oct 30 22:28:52.953838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.953809 4292 generic.go:356] "Generic (PLEG): container finished" podID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerID="95873856803e048c64e9c95fca879a785a8157a1a781798a9bd8e3faf9a694c3" exitCode=0 Oct 30 22:28:52.954105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.954023 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" event={"ID":"6a986428-4dd2-4f80-ad35-e553b64c2d3e","Type":"ContainerDied","Data":"95873856803e048c64e9c95fca879a785a8157a1a781798a9bd8e3faf9a694c3"} Oct 30 22:28:52.955774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.955708 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" event={"ID":"f862b947-3965-4ed3-bcfe-756bfcbb2487","Type":"ContainerStarted","Data":"03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b"} Oct 30 22:28:52.957917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.957895 4292 generic.go:356] "Generic (PLEG): container finished" podID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerID="028a8f55a2896c67004897e95140bac9d7089525d10c5dd490122b784d25b78a" exitCode=0 Oct 30 22:28:52.958020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:52.957962 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" event={"ID":"dc306715-040b-43b3-84c7-4c3eb267d5a0","Type":"ContainerDied","Data":"028a8f55a2896c67004897e95140bac9d7089525d10c5dd490122b784d25b78a"} Oct 30 22:28:53.000629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.000601 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:28:53.137777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.137747 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:28:53.184391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.184304 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\" (UniqueName: \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\") pod \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " Oct 30 22:28:53.184555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.184447 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\") pod \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\" (UID: \"6a986428-4dd2-4f80-ad35-e553b64c2d3e\") " Oct 30 22:28:53.184890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.184850 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume" pod "6a986428-4dd2-4f80-ad35-e553b64c2d3e" (UID: "6a986428-4dd2-4f80-ad35-e553b64c2d3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:53.185130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.185102 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a986428-4dd2-4f80-ad35-e553b64c2d3e-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.187028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.187006 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r" pod "6a986428-4dd2-4f80-ad35-e553b64c2d3e" (UID: "6a986428-4dd2-4f80-ad35-e553b64c2d3e"). InnerVolumeSpecName "kube-api-access-z7m5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:53.285700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.285666 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\" (UniqueName: \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\") pod \"dc306715-040b-43b3-84c7-4c3eb267d5a0\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " Oct 30 22:28:53.285892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.285815 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\") pod \"dc306715-040b-43b3-84c7-4c3eb267d5a0\" (UID: \"dc306715-040b-43b3-84c7-4c3eb267d5a0\") " Oct 30 22:28:53.286665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.286630 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7m5r\" (UniqueName: \"kubernetes.io/projected/6a986428-4dd2-4f80-ad35-e553b64c2d3e-kube-api-access-z7m5r\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.286982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.286954 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume" pod "dc306715-040b-43b3-84c7-4c3eb267d5a0" (UID: "dc306715-040b-43b3-84c7-4c3eb267d5a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:53.289887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.289859 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf" pod "dc306715-040b-43b3-84c7-4c3eb267d5a0" (UID: "dc306715-040b-43b3-84c7-4c3eb267d5a0"). InnerVolumeSpecName "kube-api-access-pnscf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:53.387848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.387816 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnscf\" (UniqueName: \"kubernetes.io/projected/dc306715-040b-43b3-84c7-4c3eb267d5a0-kube-api-access-pnscf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.387848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.387848 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc306715-040b-43b3-84c7-4c3eb267d5a0-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.969223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.969189 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" Oct 30 22:28:53.969689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.969189 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6" event={"ID":"6a986428-4dd2-4f80-ad35-e553b64c2d3e","Type":"ContainerDied","Data":"43c219e6f3b5022227d6b4bf283ec3f3f6c8182d94b2bb33a50d5f4acfade445"} Oct 30 22:28:53.969689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.969378 4292 scope.go:122] "RemoveContainer" containerID="95873856803e048c64e9c95fca879a785a8157a1a781798a9bd8e3faf9a694c3" Oct 30 22:28:53.971175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.971144 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" event={"ID":"f862b947-3965-4ed3-bcfe-756bfcbb2487","Type":"ContainerStarted","Data":"e70b6accf688eed37595861e4cdd654559859b74d1607b30e3277fb80d516ca8"} Oct 30 22:28:53.972556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.972529 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" event={"ID":"dc306715-040b-43b3-84c7-4c3eb267d5a0","Type":"ContainerDied","Data":"63c44281755f6eea51069ae45bed9e8de9c92f234404b8b759020de95a95fe10"} Oct 30 22:28:53.972556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.972544 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb" Oct 30 22:28:53.982025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:53.981889 4292 scope.go:122] "RemoveContainer" containerID="028a8f55a2896c67004897e95140bac9d7089525d10c5dd490122b784d25b78a" Oct 30 22:28:54.151450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.151414 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:28:54.151642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.151467 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-2jzk6"] Oct 30 22:28:54.265801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.265698 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" podStartSLOduration=3.26568266 podStartE2EDuration="3.26568266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:54.26404311 +0000 UTC m=+2062.232862675" watchObservedRunningTime="2026-10-30 22:28:54.26568266 +0000 UTC m=+2062.234502223" Oct 30 22:28:54.384257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.384216 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:28:54.390539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.390508 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7xsrb"] Oct 30 22:28:54.982632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.982586 4292 generic.go:356] "Generic (PLEG): container finished" podID="f862b947-3965-4ed3-bcfe-756bfcbb2487" containerID="e70b6accf688eed37595861e4cdd654559859b74d1607b30e3277fb80d516ca8" exitCode=0 Oct 30 22:28:54.983096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:54.982632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" event={"ID":"f862b947-3965-4ed3-bcfe-756bfcbb2487","Type":"ContainerDied","Data":"e70b6accf688eed37595861e4cdd654559859b74d1607b30e3277fb80d516ca8"} Oct 30 22:28:55.588698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:55.588664 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:28:55.588882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:55.588773 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:28:56.148280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.147961 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" path="/var/lib/kubelet/pods/6a986428-4dd2-4f80-ad35-e553b64c2d3e/volumes" Oct 30 22:28:56.148761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.148645 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" path="/var/lib/kubelet/pods/dc306715-040b-43b3-84c7-4c3eb267d5a0/volumes" Oct 30 22:28:56.183891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.183869 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:56.231615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.231560 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\") pod \"f862b947-3965-4ed3-bcfe-756bfcbb2487\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " Oct 30 22:28:56.231764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.231674 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\") pod \"f862b947-3965-4ed3-bcfe-756bfcbb2487\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " Oct 30 22:28:56.231864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.231848 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\" (UniqueName: \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\") pod \"f862b947-3965-4ed3-bcfe-756bfcbb2487\" (UID: \"f862b947-3965-4ed3-bcfe-756bfcbb2487\") " Oct 30 22:28:56.232113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.232081 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca" pod "f862b947-3965-4ed3-bcfe-756bfcbb2487" (UID: "f862b947-3965-4ed3-bcfe-756bfcbb2487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:56.232354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.232330 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:56.233724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.233703 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca" pod "f862b947-3965-4ed3-bcfe-756bfcbb2487" (UID: "f862b947-3965-4ed3-bcfe-756bfcbb2487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:56.233909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.233889 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf" pod "f862b947-3965-4ed3-bcfe-756bfcbb2487" (UID: "f862b947-3965-4ed3-bcfe-756bfcbb2487"). InnerVolumeSpecName "kube-api-access-2rfwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:56.334065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.334034 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfwf\" (UniqueName: \"kubernetes.io/projected/f862b947-3965-4ed3-bcfe-756bfcbb2487-kube-api-access-2rfwf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:56.334065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:56.334062 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f862b947-3965-4ed3-bcfe-756bfcbb2487-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:56.641786 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:28:56.641740 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:28:57.002800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.002702 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" Oct 30 22:28:57.002800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.002737 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p" event={"ID":"f862b947-3965-4ed3-bcfe-756bfcbb2487","Type":"ContainerDied","Data":"03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b"} Oct 30 22:28:57.002800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.002798 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03bd2b41a8bae73affda403bb1a36f9916d95a861ed7b283e7f5339cb44c7c7b" Oct 30 22:28:57.004496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.004471 4292 generic.go:356] "Generic (PLEG): container finished" podID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerID="fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5" exitCode=0 Oct 30 22:28:57.004623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.004527 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerDied","Data":"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5"} Oct 30 22:28:57.004954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.004928 4292 scope.go:122] "RemoveContainer" containerID="fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5" Oct 30 22:28:57.006606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.006568 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:57.006677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.006648 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:57.006713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:57.006682 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:58.018090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:58.018053 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerStarted","Data":"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316"} Oct 30 22:28:58.705922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:58.705886 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:28:58.707332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:58.706195 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:28:58.707537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:58.707511 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 22:28:59.702189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:59.701171 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:28:59.702189 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:28:59.701229 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 22:29:00.705529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:00.705490 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.706136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:00.706105 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 22:29:01.702094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:01.701618 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.702094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:01.701696 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 22:29:02.705599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:02.703003 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.705599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:02.703096 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 22:29:03.393118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:03.393082 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:29:03.506711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:03.506678 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:29:03.708785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:03.708694 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:29:03.709522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:03.709497 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:29:03.710397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:03.710375 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:29:08.394320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.394277 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" podUID="b8acacac-90b2-4685-b150-844de8d2aed8" containerName="payload-processing" containerID="cri-o://2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab" gracePeriod=25 Oct 30 22:29:08.507979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.507933 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" podUID="80d835a0-2434-4eda-927e-10b5a984c28c" containerName="payload-pre-processing" containerID="cri-o://72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2" gracePeriod=25 Oct 30 22:29:08.675538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.675511 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:29:08.761502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.761463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\") pod \"b8acacac-90b2-4685-b150-844de8d2aed8\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " Oct 30 22:29:08.761867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.761840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\" (UniqueName: \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\") pod \"b8acacac-90b2-4685-b150-844de8d2aed8\" (UID: \"b8acacac-90b2-4685-b150-844de8d2aed8\") " Oct 30 22:29:08.762101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.762076 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume" pod "b8acacac-90b2-4685-b150-844de8d2aed8" (UID: "b8acacac-90b2-4685-b150-844de8d2aed8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:08.762457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.762433 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8acacac-90b2-4685-b150-844de8d2aed8-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:08.766455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.766415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr" pod "b8acacac-90b2-4685-b150-844de8d2aed8" (UID: "b8acacac-90b2-4685-b150-844de8d2aed8"). InnerVolumeSpecName "kube-api-access-zs7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:08.785305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.785286 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:29:08.862988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.862961 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\") pod \"80d835a0-2434-4eda-927e-10b5a984c28c\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " Oct 30 22:29:08.863158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.863027 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\" (UniqueName: \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\") pod \"80d835a0-2434-4eda-927e-10b5a984c28c\" (UID: \"80d835a0-2434-4eda-927e-10b5a984c28c\") " Oct 30 22:29:08.863338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.863314 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume" pod "80d835a0-2434-4eda-927e-10b5a984c28c" (UID: "80d835a0-2434-4eda-927e-10b5a984c28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:08.863437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.863423 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d835a0-2434-4eda-927e-10b5a984c28c-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:08.863481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.863440 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7rr\" (UniqueName: \"kubernetes.io/projected/b8acacac-90b2-4685-b150-844de8d2aed8-kube-api-access-zs7rr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:08.864959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.864936 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc" pod "80d835a0-2434-4eda-927e-10b5a984c28c" (UID: "80d835a0-2434-4eda-927e-10b5a984c28c"). InnerVolumeSpecName "kube-api-access-jglmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:08.965237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:08.965162 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jglmc\" (UniqueName: \"kubernetes.io/projected/80d835a0-2434-4eda-927e-10b5a984c28c-kube-api-access-jglmc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.100865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.100828 4292 generic.go:356] "Generic (PLEG): container finished" podID="80d835a0-2434-4eda-927e-10b5a984c28c" containerID="72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2" exitCode=0 Oct 30 22:29:09.101017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.100877 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" Oct 30 22:29:09.101017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.100912 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" event={"ID":"80d835a0-2434-4eda-927e-10b5a984c28c","Type":"ContainerDied","Data":"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2"} Oct 30 22:29:09.101017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.100967 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw" event={"ID":"80d835a0-2434-4eda-927e-10b5a984c28c","Type":"ContainerDied","Data":"665612cba9a415e0f14503a3ceefca41d70072868c93c31b721ebf37b51c13ec"} Oct 30 22:29:09.101017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.100993 4292 scope.go:122] "RemoveContainer" containerID="72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2" Oct 30 22:29:09.102451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.102428 4292 generic.go:356] "Generic (PLEG): container finished" podID="b8acacac-90b2-4685-b150-844de8d2aed8" containerID="2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab" exitCode=0 Oct 30 22:29:09.102564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.102477 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" Oct 30 22:29:09.102564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.102504 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" event={"ID":"b8acacac-90b2-4685-b150-844de8d2aed8","Type":"ContainerDied","Data":"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab"} Oct 30 22:29:09.102789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.102604 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg" event={"ID":"b8acacac-90b2-4685-b150-844de8d2aed8","Type":"ContainerDied","Data":"21bb403ea12ff8d8fb68b3473edcad6567d6d662c4977bd8381f69ee053c437d"} Oct 30 22:29:09.112229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.112208 4292 scope.go:122] "RemoveContainer" containerID="72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2" Oct 30 22:29:09.112484 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:09.112465 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2\": container with ID starting with 72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2 not found: ID does not exist" containerID="72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2" Oct 30 22:29:09.112558 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.112489 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2"} err="failed to get container status \"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2\": rpc error: code = NotFound desc = could not find container \"72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2\": container with ID starting with 72357c28f55293131c3f13d15f99d24313d356b98ab800e75d4291e9920d27f2 not found: ID does not exist" Oct 30 22:29:09.112558 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.112507 4292 scope.go:122] "RemoveContainer" containerID="2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab" Oct 30 22:29:09.122186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.122169 4292 scope.go:122] "RemoveContainer" containerID="2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab" Oct 30 22:29:09.122429 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:09.122411 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab\": container with ID starting with 2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab not found: ID does not exist" containerID="2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab" Oct 30 22:29:09.122478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.122437 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab"} err="failed to get container status \"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab\": rpc error: code = NotFound desc = could not find container \"2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab\": container with ID starting with 2df9dba3c6302a36f6163f457780ddda6ec2c1a1cbcec4aa38a2cf88226e8cab not found: ID does not exist" Oct 30 22:29:09.283143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.283083 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:29:09.287969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.287942 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afe802-78f5df8bb7-c85xg"] Oct 30 22:29:09.401245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.401216 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:29:09.411319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:09.411283 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afe802-779fc4d8d7-s54kw"] Oct 30 22:29:10.148588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:10.148553 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d835a0-2434-4eda-927e-10b5a984c28c" path="/var/lib/kubelet/pods/80d835a0-2434-4eda-927e-10b5a984c28c/volumes" Oct 30 22:29:10.148985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:10.148971 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8acacac-90b2-4685-b150-844de8d2aed8" path="/var/lib/kubelet/pods/b8acacac-90b2-4685-b150-844de8d2aed8/volumes" Oct 30 22:29:14.369482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.369448 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:29:14.370016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.369657 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerName="discovery" containerID="cri-o://bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e" gracePeriod=30 Oct 30 22:29:14.647845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.647820 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:29:14.741563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741537 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.741725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741616 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.741725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741680 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.741725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741712 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.741899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741791 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.741899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.741877 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.742061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.742032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.742125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.742068 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\") pod \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\" (UID: \"1687ba16-2f7f-42d3-96c9-2faa5e1be479\") " Oct 30 22:29:14.742803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.742778 4292 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-ca-configmap\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.744177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744153 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:14.744620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744552 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:14.744730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744624 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.744730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744683 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:14.744819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744742 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "kube-api-access-vsd5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.744980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.744966 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token" pod "1687ba16-2f7f-42d3-96c9-2faa5e1be479" (UID: "1687ba16-2f7f-42d3-96c9-2faa5e1be479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877693 4292 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-cacerts\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877742 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsd5n\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-kube-api-access-vsd5n\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877760 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877777 4292 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-csr-dns-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877792 4292 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1687ba16-2f7f-42d3-96c9-2faa5e1be479-local-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.877805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:14.877805 4292 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1687ba16-2f7f-42d3-96c9-2faa5e1be479-istio-kubeconfig\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:15.148993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.148912 4292 generic.go:356] "Generic (PLEG): container finished" podID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerID="bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e" exitCode=0 Oct 30 22:29:15.148993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.148969 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" Oct 30 22:29:15.149211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.148989 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" event={"ID":"1687ba16-2f7f-42d3-96c9-2faa5e1be479","Type":"ContainerDied","Data":"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e"} Oct 30 22:29:15.149211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.149033 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk" event={"ID":"1687ba16-2f7f-42d3-96c9-2faa5e1be479","Type":"ContainerDied","Data":"8640973ba7975ff4607102c2242577d2a7f76f05caa167571674b95e888a7358"} Oct 30 22:29:15.149211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.149054 4292 scope.go:122] "RemoveContainer" containerID="bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e" Oct 30 22:29:15.159995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.159977 4292 scope.go:122] "RemoveContainer" containerID="bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e" Oct 30 22:29:15.160241 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:15.160224 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e\": container with ID starting with bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e not found: ID does not exist" containerID="bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e" Oct 30 22:29:15.160284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.160250 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e"} err="failed to get container status \"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e\": rpc error: code = NotFound desc = could not find container \"bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e\": container with ID starting with bd825f47a2e85e4bbb9141db87da06e932efd04f595b4c00a4edaa9f9fd1607e not found: ID does not exist" Oct 30 22:29:15.322191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.322158 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:29:15.372382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:15.372358 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lb4wk"] Oct 30 22:29:16.148005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:16.147970 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" path="/var/lib/kubelet/pods/1687ba16-2f7f-42d3-96c9-2faa5e1be479/volumes" Oct 30 22:29:21.541246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:21.541202 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:29:21.541783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:21.541483 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" containerID="cri-o://d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4" gracePeriod=30 Oct 30 22:29:24.478299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:24.478262 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:24.478824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:24.478329 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:25.285851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.285700 4292 generic.go:356] "Generic (PLEG): container finished" podID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerID="9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742" exitCode=0 Oct 30 22:29:25.285851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.285778 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerDied","Data":"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742"} Oct 30 22:29:25.286203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.286182 4292 scope.go:122] "RemoveContainer" containerID="9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742" Oct 30 22:29:25.287826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.287795 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:25.287933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.287865 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:25.287933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:25.287898 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:26.297072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.297039 4292 generic.go:356] "Generic (PLEG): container finished" podID="5ce900df-26aa-4630-8bca-bb946483a843" containerID="6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389" exitCode=0 Oct 30 22:29:26.297622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.297111 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerDied","Data":"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389"} Oct 30 22:29:26.297622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.297608 4292 scope.go:122] "RemoveContainer" containerID="6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389" Oct 30 22:29:26.300390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.299687 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerStarted","Data":"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6"} Oct 30 22:29:26.300390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.300016 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:26.300390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.300116 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:26.300390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.300172 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:26.644603 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:26.644528 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:29:26.750617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.749353 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:29:26.750617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.749530 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:26.750617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.749602 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:26.863079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.863058 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:29:26.940350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940314 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940441 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940489 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940600 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:26.940690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940695 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940828 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.940868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940847 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:26.941027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940855 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.941027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940927 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:26.941027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.940965 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:26.941027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941004 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\") pod \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\" (UID: \"029eb867-dcc6-45e5-a2c2-1798192b4f4d\") " Oct 30 22:29:26.941336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941311 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:26.941675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941655 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:26.941675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941675 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:26.941832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941685 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:26.941832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941693 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:26.941832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.941703 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:26.942778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.942753 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:26.943038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.943020 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:26.943317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.943291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:26.943416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:26.943320 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg" pod "029eb867-dcc6-45e5-a2c2-1798192b4f4d" (UID: "029eb867-dcc6-45e5-a2c2-1798192b4f4d"). InnerVolumeSpecName "kube-api-access-gv7tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:27.042999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.042958 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.043167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.043013 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.043167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.043030 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7tg\" (UniqueName: \"kubernetes.io/projected/029eb867-dcc6-45e5-a2c2-1798192b4f4d-kube-api-access-gv7tg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.043167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.043047 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029eb867-dcc6-45e5-a2c2-1798192b4f4d-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.338632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.337187 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerStarted","Data":"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1"} Oct 30 22:29:27.339379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.339352 4292 generic.go:356] "Generic (PLEG): container finished" podID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerID="d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4" exitCode=0 Oct 30 22:29:27.339517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.339415 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" event={"ID":"029eb867-dcc6-45e5-a2c2-1798192b4f4d","Type":"ContainerDied","Data":"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4"} Oct 30 22:29:27.339517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.339449 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" Oct 30 22:29:27.339517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.339474 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss" event={"ID":"029eb867-dcc6-45e5-a2c2-1798192b4f4d","Type":"ContainerDied","Data":"b1435d57c66a1f0ee5b68deb31fe31a66aefe3c57a8a825d02912f40f81c1c06"} Oct 30 22:29:27.339517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.339501 4292 scope.go:122] "RemoveContainer" containerID="d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4" Oct 30 22:29:27.353653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.353635 4292 scope.go:122] "RemoveContainer" containerID="d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4" Oct 30 22:29:27.353889 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:27.353868 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4\": container with ID starting with d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4 not found: ID does not exist" containerID="d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4" Oct 30 22:29:27.353946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.353898 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4"} err="failed to get container status \"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4\": rpc error: code = NotFound desc = could not find container \"d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4\": container with ID starting with d4ecd3aa8f30be8165b948504a45d61a9dc54b021bbb1696160cbc975735a1f4 not found: ID does not exist" Oct 30 22:29:27.640658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.640186 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:29:27.648913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.647645 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-hvkss"] Oct 30 22:29:27.745824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.745481 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:27.745824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:27.745540 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:28.155658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.155618 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" path="/var/lib/kubelet/pods/029eb867-dcc6-45e5-a2c2-1798192b4f4d/volumes" Oct 30 22:29:28.183633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.183607 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:29:28.192601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.184615 4292 patch_prober.go:29] interesting pod/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:28.192601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.184673 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 22:29:28.750594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.748922 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:28.750594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.748984 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:28.820716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:28.820681 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:29:29.183965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.183935 4292 patch_prober.go:29] interesting pod/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:29.184195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.184174 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 22:29:29.759601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.756664 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:29.759601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.756725 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:29.934597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.934226 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk"] Oct 30 22:29:29.942747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:29.942722 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a42692-msblk"] Oct 30 22:29:30.148471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:30.148435 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0a5a59-22ca-4089-a2fc-591b018f6bd9" path="/var/lib/kubelet/pods/fa0a5a59-22ca-4089-a2fc-591b018f6bd9/volumes" Oct 30 22:29:30.183737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:30.183710 4292 patch_prober.go:29] interesting pod/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:30.183864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:30.183764 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 22:29:30.744674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:30.744635 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:30.744877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:30.744693 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:31.202628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:31.200658 4292 patch_prober.go:29] interesting pod/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:31.202628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:31.200729 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 22:29:31.746614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:31.746558 4292 patch_prober.go:29] interesting pod/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 22:29:31.746798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:31.746635 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 22:29:32.183799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:32.183766 4292 patch_prober.go:29] interesting pod/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:32.184002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:32.183811 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 22:29:32.749515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:32.749483 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:29:32.750008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:32.749989 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:29:32.750760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:32.750739 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:29:33.189031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:33.188999 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:29:33.190198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:33.190178 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:29:37.083272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:37.083239 4292 scope.go:122] "RemoveContainer" containerID="74a7c135ac2e50c97dcf1701fd2d68690923f9f6650e130535cb82c5f1dd037b" Oct 30 22:29:37.093122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:37.093106 4292 scope.go:122] "RemoveContainer" containerID="660ffaa2393a1fd303036766f12eabd600da1103db8a3ee3c64077b97744d7ea" Oct 30 22:29:40.228354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.228324 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44"] Oct 30 22:29:40.228978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.228962 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" Oct 30 22:29:40.229024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.228979 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" Oct 30 22:29:40.229024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.228988 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" Oct 30 22:29:40.229024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.228993 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" Oct 30 22:29:40.229024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229002 4292 cpu_manager.go:414] "containerMap: removing container" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" Oct 30 22:29:40.229024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229007 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229037 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f862b947-3965-4ed3-bcfe-756bfcbb2487" containerName="revoke-keys" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229042 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f862b947-3965-4ed3-bcfe-756bfcbb2487" containerName="revoke-keys" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="80d835a0-2434-4eda-927e-10b5a984c28c" containerName="payload-pre-processing" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229057 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d835a0-2434-4eda-927e-10b5a984c28c" containerName="payload-pre-processing" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229079 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b8acacac-90b2-4685-b150-844de8d2aed8" containerName="payload-processing" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229085 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8acacac-90b2-4685-b150-844de8d2aed8" containerName="payload-processing" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229095 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerName="discovery" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229099 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerName="discovery" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229180 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8acacac-90b2-4685-b150-844de8d2aed8" containerName="payload-processing" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229190 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1687ba16-2f7f-42d3-96c9-2faa5e1be479" containerName="discovery" Oct 30 22:29:40.229194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229199 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc306715-040b-43b3-84c7-4c3eb267d5a0" containerName="payload-pre-processing" Oct 30 22:29:40.229545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229206 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a986428-4dd2-4f80-ad35-e553b64c2d3e" containerName="payload-processing" Oct 30 22:29:40.229545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229211 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="029eb867-dcc6-45e5-a2c2-1798192b4f4d" containerName="istio-proxy" Oct 30 22:29:40.229545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229220 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f862b947-3965-4ed3-bcfe-756bfcbb2487" containerName="revoke-keys" Oct 30 22:29:40.229545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.229227 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d835a0-2434-4eda-927e-10b5a984c28c" containerName="payload-pre-processing" Oct 30 22:29:40.234679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.234659 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.283625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.283600 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbf6\" (UniqueName: \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.283804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.283782 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.283906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.283891 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.291664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.291631 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44"] Oct 30 22:29:40.337695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.337661 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:29:40.337695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.337662 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:29:40.337838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.337676 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:29:40.338723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.338703 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:29:40.385163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.385136 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.385250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.385190 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbf6\" (UniqueName: \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.385331 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.385318 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.385925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.385901 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.387451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.387430 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.453954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.453930 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbf6\" (UniqueName: \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.548930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.548905 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:40.938761 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:29:40.938727 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cc9422_0eff_481c_9832_0e795a768aff.slice/crio-bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101 WatchSource:0}: Error finding container bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101: Status 404 returned error can't find the container with id bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101 Oct 30 22:29:40.939076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:40.939052 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44"] Oct 30 22:29:41.386115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.386084 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw"] Oct 30 22:29:41.390381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.390362 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.449657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.449632 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw"] Oct 30 22:29:41.462675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.462610 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" event={"ID":"16cc9422-0eff-481c-9832-0e795a768aff","Type":"ContainerStarted","Data":"d41faffa138d359c21271687cec3c89de20ae8bb6fd50195a351862673e9a3cd"} Oct 30 22:29:41.462675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.462666 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" event={"ID":"16cc9422-0eff-481c-9832-0e795a768aff","Type":"ContainerStarted","Data":"bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101"} Oct 30 22:29:41.499695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.499672 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlfl\" (UniqueName: \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.499783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.499706 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.499783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.499729 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.600948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.600913 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlfl\" (UniqueName: \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.601107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.600962 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.601107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.600989 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.601726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.601706 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.603349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.603327 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.643016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.642971 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" podStartSLOduration=1.642958366 podStartE2EDuration="1.642958366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:41.63854276 +0000 UTC m=+2109.607362324" watchObservedRunningTime="2026-10-30 22:29:41.642958366 +0000 UTC m=+2109.611777930" Oct 30 22:29:41.669997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.669977 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlfl\" (UniqueName: \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:41.718098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:41.718029 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:42.102475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.102450 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw"] Oct 30 22:29:42.103104 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:29:42.103076 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6629d24_3d90_47b3_9b9b_9f4251cd2818.slice/crio-1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f WatchSource:0}: Error finding container 1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f: Status 404 returned error can't find the container with id 1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f Oct 30 22:29:42.483431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.483395 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" event={"ID":"d6629d24-3d90-47b3-9b9b-9f4251cd2818","Type":"ContainerStarted","Data":"85403ddd395fb766f8a96f793f068067157ca56ccb54ceb2c66e8b4689d4a476"} Oct 30 22:29:42.483849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.483481 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" event={"ID":"d6629d24-3d90-47b3-9b9b-9f4251cd2818","Type":"ContainerStarted","Data":"1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f"} Oct 30 22:29:42.485099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.485071 4292 generic.go:356] "Generic (PLEG): container finished" podID="16cc9422-0eff-481c-9832-0e795a768aff" containerID="d41faffa138d359c21271687cec3c89de20ae8bb6fd50195a351862673e9a3cd" exitCode=0 Oct 30 22:29:42.485188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.485141 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" event={"ID":"16cc9422-0eff-481c-9832-0e795a768aff","Type":"ContainerDied","Data":"d41faffa138d359c21271687cec3c89de20ae8bb6fd50195a351862673e9a3cd"} Oct 30 22:29:42.605682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:42.605637 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" podStartSLOduration=1.60562355 podStartE2EDuration="1.60562355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:42.602441309 +0000 UTC m=+2110.571260898" watchObservedRunningTime="2026-10-30 22:29:42.60562355 +0000 UTC m=+2110.574443114" Oct 30 22:29:43.492976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.492941 4292 generic.go:356] "Generic (PLEG): container finished" podID="d6629d24-3d90-47b3-9b9b-9f4251cd2818" containerID="85403ddd395fb766f8a96f793f068067157ca56ccb54ceb2c66e8b4689d4a476" exitCode=0 Oct 30 22:29:43.493427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.492984 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" event={"ID":"d6629d24-3d90-47b3-9b9b-9f4251cd2818","Type":"ContainerDied","Data":"85403ddd395fb766f8a96f793f068067157ca56ccb54ceb2c66e8b4689d4a476"} Oct 30 22:29:43.662641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.662612 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:43.727382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.727354 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\") pod \"16cc9422-0eff-481c-9832-0e795a768aff\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " Oct 30 22:29:43.727546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.727501 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\" (UniqueName: \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\") pod \"16cc9422-0eff-481c-9832-0e795a768aff\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " Oct 30 22:29:43.727682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.727656 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\") pod \"16cc9422-0eff-481c-9832-0e795a768aff\" (UID: \"16cc9422-0eff-481c-9832-0e795a768aff\") " Oct 30 22:29:43.727965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.727939 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca" pod "16cc9422-0eff-481c-9832-0e795a768aff" (UID: "16cc9422-0eff-481c-9832-0e795a768aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:43.728154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.728139 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:43.729710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.729687 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6" pod "16cc9422-0eff-481c-9832-0e795a768aff" (UID: "16cc9422-0eff-481c-9832-0e795a768aff"). InnerVolumeSpecName "kube-api-access-sdbf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:43.729776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.729735 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca" pod "16cc9422-0eff-481c-9832-0e795a768aff" (UID: "16cc9422-0eff-481c-9832-0e795a768aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:43.829513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.829476 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc9422-0eff-481c-9832-0e795a768aff-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:43.829513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:43.829515 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbf6\" (UniqueName: \"kubernetes.io/projected/16cc9422-0eff-481c-9832-0e795a768aff-kube-api-access-sdbf6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:44.501994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.501869 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" Oct 30 22:29:44.501994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.501906 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44" event={"ID":"16cc9422-0eff-481c-9832-0e795a768aff","Type":"ContainerDied","Data":"bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101"} Oct 30 22:29:44.501994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.501969 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef3b9d77a2d0ba69584baf618a76de305f94dbf4f3279c05a6792b899cef101" Oct 30 22:29:44.652660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.652636 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:44.740434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.740405 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\") pod \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " Oct 30 22:29:44.740619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.740463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\") pod \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " Oct 30 22:29:44.740698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.740622 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\" (UniqueName: \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\") pod \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\" (UID: \"d6629d24-3d90-47b3-9b9b-9f4251cd2818\") " Oct 30 22:29:44.740878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.740852 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca" pod "d6629d24-3d90-47b3-9b9b-9f4251cd2818" (UID: "d6629d24-3d90-47b3-9b9b-9f4251cd2818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:44.741192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.741174 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:44.742702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.742679 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca" pod "d6629d24-3d90-47b3-9b9b-9f4251cd2818" (UID: "d6629d24-3d90-47b3-9b9b-9f4251cd2818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:44.742809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.742719 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl" pod "d6629d24-3d90-47b3-9b9b-9f4251cd2818" (UID: "d6629d24-3d90-47b3-9b9b-9f4251cd2818"). InnerVolumeSpecName "kube-api-access-7mlfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:44.842405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.842369 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6629d24-3d90-47b3-9b9b-9f4251cd2818-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:44.842405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:44.842403 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlfl\" (UniqueName: \"kubernetes.io/projected/d6629d24-3d90-47b3-9b9b-9f4251cd2818-kube-api-access-7mlfl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:45.510191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:45.510160 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" Oct 30 22:29:45.510556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:45.510190 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw" event={"ID":"d6629d24-3d90-47b3-9b9b-9f4251cd2818","Type":"ContainerDied","Data":"1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f"} Oct 30 22:29:45.510556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:45.510239 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e8eabd74cfd891c91efe33ebaef9442711728be38410e1a74b2e6d5d7642b6f" Oct 30 22:29:45.645425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:45.645383 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:29:52.418741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:52.418702 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:29:52.528595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:52.528513 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:29:54.896530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:54.896492 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:29:55.008006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:55.007631 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:29:56.668332 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:29:56.668289 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:29:57.419605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:57.419548 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" containerID="cri-o://f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572" gracePeriod=25 Oct 30 22:29:57.529225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:57.529187 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" containerID="cri-o://827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0" gracePeriod=25 Oct 30 22:29:59.897568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:29:59.897528 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" podUID="10882900-cceb-48dd-bf40-486a132bed75" containerName="payload-processing" containerID="cri-o://dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00" gracePeriod=25 Oct 30 22:30:00.008475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.008433 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" podUID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" containerName="payload-pre-processing" containerID="cri-o://f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231" gracePeriod=25 Oct 30 22:30:00.178893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.178862 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:30:00.181206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181175 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:00.181886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181866 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d6629d24-3d90-47b3-9b9b-9f4251cd2818" containerName="revoke-keys" Oct 30 22:30:00.181999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181888 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6629d24-3d90-47b3-9b9b-9f4251cd2818" containerName="revoke-keys" Oct 30 22:30:00.181999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181910 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16cc9422-0eff-481c-9832-0e795a768aff" containerName="revoke-keys" Oct 30 22:30:00.181999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181917 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cc9422-0eff-481c-9832-0e795a768aff" containerName="revoke-keys" Oct 30 22:30:00.181999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181949 4292 cpu_manager.go:414] "containerMap: removing container" podUID="10882900-cceb-48dd-bf40-486a132bed75" containerName="payload-processing" Oct 30 22:30:00.181999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.181956 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10882900-cceb-48dd-bf40-486a132bed75" containerName="payload-processing" Oct 30 22:30:00.182297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.182098 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6629d24-3d90-47b3-9b9b-9f4251cd2818" containerName="revoke-keys" Oct 30 22:30:00.182297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.182108 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cc9422-0eff-481c-9832-0e795a768aff" containerName="revoke-keys" Oct 30 22:30:00.182297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.182120 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="10882900-cceb-48dd-bf40-486a132bed75" containerName="payload-processing" Oct 30 22:30:00.185642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.185621 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:00.185880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.185853 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.190634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.190567 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:00.190739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.190714 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.195601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.194774 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.234531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.234460 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:00.251783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.251744 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:00.252541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.252054 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.256552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.256522 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\") pod \"10882900-cceb-48dd-bf40-486a132bed75\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " Oct 30 22:30:00.256982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.256957 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\" (UniqueName: \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\") pod \"10882900-cceb-48dd-bf40-486a132bed75\" (UID: \"10882900-cceb-48dd-bf40-486a132bed75\") " Oct 30 22:30:00.257436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.257412 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxcj\" (UniqueName: \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.257655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.257635 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.257824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.257804 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9wh\" (UniqueName: \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.257972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.257954 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4tqk\" (UniqueName: \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.258922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.258900 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.259116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.259097 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.259222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.259196 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume" pod "10882900-cceb-48dd-bf40-486a132bed75" (UID: "10882900-cceb-48dd-bf40-486a132bed75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:00.259553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.259535 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10882900-cceb-48dd-bf40-486a132bed75-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:00.270290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.262711 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r" pod "10882900-cceb-48dd-bf40-486a132bed75" (UID: "10882900-cceb-48dd-bf40-486a132bed75"). InnerVolumeSpecName "kube-api-access-ptr9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:00.270290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.266225 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:00.270290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.266330 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.270564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.270546 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:00.270711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.270687 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.274802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.274782 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:00.274923 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.274907 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.280372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.280349 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:00.280665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.280633 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.281976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.281957 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:30:00.282094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.281991 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.282203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.282125 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.286175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.286153 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.289308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.289287 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:00.293524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.293503 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:00.298186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.297354 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:00.304248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.304224 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:00.304477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.304453 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.316764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.316738 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:00.317055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.317039 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.318659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.318636 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:00.323613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.323135 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:00.356291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.356263 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:00.356628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.356609 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:00.356804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.356542 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.362631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.362088 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:00.363001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.362849 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6m2\" (UniqueName: \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.363001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.362885 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5pn\" (UniqueName: \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.363001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.362911 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.363001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.362990 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxcj\" (UniqueName: \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.363267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363219 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jvh\" (UniqueName: \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.363335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363280 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.363335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9wh\" (UniqueName: \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363348 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4tqk\" (UniqueName: \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcn5j\" (UniqueName: \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363549 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363567 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrjb\" (UniqueName: \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363622 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5lj\" (UniqueName: \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363653 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363747 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.363783 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.364494 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.364618 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.364718 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.364865 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.364869 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr9r\" (UniqueName: \"kubernetes.io/projected/10882900-cceb-48dd-bf40-486a132bed75-kube-api-access-ptr9r\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:00.366437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.365556 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.367379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.367264 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.373885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.372666 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:00.400633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.391161 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:00.406790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.406642 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:00.408391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.408364 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:30:00.414239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.414215 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:00.427720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.427699 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxcj\" (UniqueName: \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\") pod \"maas-api-key-cleanup-29889990-x4k9n\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.428714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.428505 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4tqk\" (UniqueName: \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.429046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.429020 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9wh\" (UniqueName: \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\") pod \"maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.452724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.452694 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:00.466490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466463 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.466663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466541 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.466663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466609 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxbt\" (UniqueName: \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.466663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466648 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6m2\" (UniqueName: \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.466855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466680 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5pn\" (UniqueName: \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.466855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466704 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbsf\" (UniqueName: \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.466855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466801 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.467018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466852 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jvh\" (UniqueName: \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.467018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466896 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.467018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.466994 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lpj\" (UniqueName: \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.467184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467042 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcn5j\" (UniqueName: \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.467184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467110 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.467184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467139 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrjb\" (UniqueName: \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.467343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467182 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5lj\" (UniqueName: \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.467343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.467343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467316 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.467343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467325 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.467600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467348 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.467600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467426 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.467600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467517 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.468373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.467966 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.468373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.468352 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.468688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.468665 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.469218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.469193 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.519771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.519740 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:00.530424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.530167 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:00.536400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.536375 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:00.540949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.540921 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5lj\" (UniqueName: \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\") pod \"maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.541064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.540979 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6m2\" (UniqueName: \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.542334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.542308 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcn5j\" (UniqueName: \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.542334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.542323 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jvh\" (UniqueName: \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.542539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.542363 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrjb\" (UniqueName: \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\") pod \"maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.544283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.544263 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5pn\" (UniqueName: \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\") pod \"maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569077 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\" (UniqueName: \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\") pod \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569328 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\") pod \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\" (UID: \"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0\") " Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569687 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxbt\" (UniqueName: \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569743 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbsf\" (UniqueName: \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569797 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569874 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lpj\" (UniqueName: \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.569969 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.570289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.570032 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.572100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.572072 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.572788 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.572764 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume" pod "d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" (UID: "d6441d57-ca7e-4f63-9dc5-6e0088bd81c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:00.573201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.573178 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.573359 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.573340 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc" pod "d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" (UID: "d6441d57-ca7e-4f63-9dc5-6e0088bd81c0"). InnerVolumeSpecName "kube-api-access-hzjkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:00.573854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.573833 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.590071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.590043 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:00.608659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.603829 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:00.624283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.622912 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:00.639136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.639109 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lpj\" (UniqueName: \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\") pod \"maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.639856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.639832 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbsf\" (UniqueName: \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.640764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.640743 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxbt\" (UniqueName: \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\") pod \"maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.653089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.653068 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:00.669626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.669378 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:00.673596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.672139 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjkc\" (UniqueName: \"kubernetes.io/projected/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-kube-api-access-hzjkc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:00.673596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.672170 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:00.677731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.676813 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" containerID="cri-o://0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c" gracePeriod=45 Oct 30 22:30:00.678230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.678182 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:00.716688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.716028 4292 generic.go:356] "Generic (PLEG): container finished" podID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" containerID="f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231" exitCode=0 Oct 30 22:30:00.716688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.716118 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" event={"ID":"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0","Type":"ContainerDied","Data":"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231"} Oct 30 22:30:00.716688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.716172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" event={"ID":"d6441d57-ca7e-4f63-9dc5-6e0088bd81c0","Type":"ContainerDied","Data":"29939cdbe5e38b9b1d01ffd07446a3717a1bfc3e6ea0085ef862a773eab78e37"} Oct 30 22:30:00.716688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.716201 4292 scope.go:122] "RemoveContainer" containerID="f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231" Oct 30 22:30:00.716688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.716370 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr" Oct 30 22:30:00.725150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.724848 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:00.732835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.732805 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:00.741894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.741547 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:00.763508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.762448 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" Oct 30 22:30:00.765766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.765696 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" event={"ID":"10882900-cceb-48dd-bf40-486a132bed75","Type":"ContainerDied","Data":"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00"} Oct 30 22:30:00.766002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.762301 4292 generic.go:356] "Generic (PLEG): container finished" podID="10882900-cceb-48dd-bf40-486a132bed75" containerID="dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00" exitCode=0 Oct 30 22:30:00.766002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.765960 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8" event={"ID":"10882900-cceb-48dd-bf40-486a132bed75","Type":"ContainerDied","Data":"e6431651144bac49ee41f3735804f8481e0bd9db8f607c5b5b847d667bd305f4"} Oct 30 22:30:00.857883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.857755 4292 scope.go:122] "RemoveContainer" containerID="f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231" Oct 30 22:30:00.859149 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:00.858984 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231\": container with ID starting with f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231 not found: ID does not exist" containerID="f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231" Oct 30 22:30:00.859149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.859026 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231"} err="failed to get container status \"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231\": rpc error: code = NotFound desc = could not find container \"f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231\": container with ID starting with f938bde3f14695c8e5b5872a9028e0b98ca655987efba2f407e4cc1ad5df3231 not found: ID does not exist" Oct 30 22:30:00.859149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.859049 4292 scope.go:122] "RemoveContainer" containerID="dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00" Oct 30 22:30:00.882770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.882741 4292 scope.go:122] "RemoveContainer" containerID="dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00" Oct 30 22:30:00.883158 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:00.883130 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00\": container with ID starting with dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00 not found: ID does not exist" containerID="dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00" Oct 30 22:30:00.883218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.883172 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00"} err="failed to get container status \"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00\": rpc error: code = NotFound desc = could not find container \"dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00\": container with ID starting with dbed5d5cad5414c8ecb35bdbde834a9fcea4e4ca41d017c8aef23231850c1f00 not found: ID does not exist" Oct 30 22:30:00.953848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:00.953811 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:01.160061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.160033 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:30:01.289354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.289324 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\" (UniqueName: \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.289854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.289833 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.290013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.289922 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.290105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.290017 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.290105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.290071 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.290369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.290141 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\") pod \"b3e077c1-045f-4768-9778-ec5f714b6183\" (UID: \"b3e077c1-045f-4768-9778-ec5f714b6183\") " Oct 30 22:30:01.291193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.291163 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.291301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.291235 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.291668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.291646 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.293768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.293743 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "kube-api-access-hrqj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:01.293917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.293760 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:01.295172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.295145 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm" pod "b3e077c1-045f-4768-9778-ec5f714b6183" (UID: "b3e077c1-045f-4768-9778-ec5f714b6183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.332804 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.332771 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12303c3_00b0_4d84_b007_a3f040b44ea6.slice/crio-1f092b1660692d13075785a86874451d5ca638bb8f6005112e8ea7e90dc72400 WatchSource:0}: Error finding container 1f092b1660692d13075785a86874451d5ca638bb8f6005112e8ea7e90dc72400: Status 404 returned error can't find the container with id 1f092b1660692d13075785a86874451d5ca638bb8f6005112e8ea7e90dc72400 Oct 30 22:30:01.333833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.333807 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:01.356493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.356444 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:01.357327 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.357295 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b99f5a_0442_454d_9e8e_a61459da0505.slice/crio-7e4050067ca7bd0b4b4bd1791493609c2f5c4a3a293e39f374cfad98b0431d1f WatchSource:0}: Error finding container 7e4050067ca7bd0b4b4bd1791493609c2f5c4a3a293e39f374cfad98b0431d1f: Status 404 returned error can't find the container with id 7e4050067ca7bd0b4b4bd1791493609c2f5c4a3a293e39f374cfad98b0431d1f Oct 30 22:30:01.380175 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.380142 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8e9159_6c6e_4259_891c_d292138bf8d3.slice/crio-92d5b53d60ead7108a08763f6e199e26a1417b37665ef190f709cc65d6b82608 WatchSource:0}: Error finding container 92d5b53d60ead7108a08763f6e199e26a1417b37665ef190f709cc65d6b82608: Status 404 returned error can't find the container with id 92d5b53d60ead7108a08763f6e199e26a1417b37665ef190f709cc65d6b82608 Oct 30 22:30:01.380527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.380507 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:01.387844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.387815 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:01.391981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.391950 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.392072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.391989 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqj5\" (UniqueName: \"kubernetes.io/projected/b3e077c1-045f-4768-9778-ec5f714b6183-kube-api-access-hrqj5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.392072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.392006 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.392072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.392020 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.392072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.392032 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e077c1-045f-4768-9778-ec5f714b6183-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.392072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.392045 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e077c1-045f-4768-9778-ec5f714b6183-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.396976 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.396952 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a67acb1_27b6_422c_9ba0_c6663b09fa46.slice/crio-ba16d485690123072f31bb4269ba9f8586c6c1092517e19be77ca80d3393b8e6 WatchSource:0}: Error finding container ba16d485690123072f31bb4269ba9f8586c6c1092517e19be77ca80d3393b8e6: Status 404 returned error can't find the container with id ba16d485690123072f31bb4269ba9f8586c6c1092517e19be77ca80d3393b8e6 Oct 30 22:30:01.445930 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.445895 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28edfe07_b99a_43e5_b724_ac304d6de0ae.slice/crio-b28770c6dcb92f2442dc1e2f6d6b8e82cd4a748ee35c96137ad50c59055f34c0 WatchSource:0}: Error finding container b28770c6dcb92f2442dc1e2f6d6b8e82cd4a748ee35c96137ad50c59055f34c0: Status 404 returned error can't find the container with id b28770c6dcb92f2442dc1e2f6d6b8e82cd4a748ee35c96137ad50c59055f34c0 Oct 30 22:30:01.447688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.447656 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:01.453975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.453949 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:01.458933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.458905 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:01.459188 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.459167 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08061f89_9348_4dca_ba8f_78d7447a875b.slice/crio-1bf472415f2be922f8d7ac63f4df31658a4aaf000cf8576a8f87d3ebaedca8ba WatchSource:0}: Error finding container 1bf472415f2be922f8d7ac63f4df31658a4aaf000cf8576a8f87d3ebaedca8ba: Status 404 returned error can't find the container with id 1bf472415f2be922f8d7ac63f4df31658a4aaf000cf8576a8f87d3ebaedca8ba Oct 30 22:30:01.461604 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.461553 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ae0e55_b704_4446_a5f8_855033b855f0.slice/crio-afddd8c6e7aa5d94df8e88eb6b5d0fb3ada922ec70e40a40f2026298f9ab02c5 WatchSource:0}: Error finding container afddd8c6e7aa5d94df8e88eb6b5d0fb3ada922ec70e40a40f2026298f9ab02c5: Status 404 returned error can't find the container with id afddd8c6e7aa5d94df8e88eb6b5d0fb3ada922ec70e40a40f2026298f9ab02c5 Oct 30 22:30:01.490097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.490063 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:01.491352 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.491312 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d2a0e3_b6dc_417f_89da_4fd53c19b981.slice/crio-feb42c16f764763767154dfcd1e40c27c65426d187c4ed83dee0feae9b96eb88 WatchSource:0}: Error finding container feb42c16f764763767154dfcd1e40c27c65426d187c4ed83dee0feae9b96eb88: Status 404 returned error can't find the container with id feb42c16f764763767154dfcd1e40c27c65426d187c4ed83dee0feae9b96eb88 Oct 30 22:30:01.496942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.496912 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:01.519537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.519513 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:01.523607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.523550 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:01.688889 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.688728 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d11fe00_6f4d_42ce_ba99_252a4b1f0dcc.slice/crio-541ea9759952decfca7da31f4a85e41fe3489c1312f850b0626850de33c43a65 WatchSource:0}: Error finding container 541ea9759952decfca7da31f4a85e41fe3489c1312f850b0626850de33c43a65: Status 404 returned error can't find the container with id 541ea9759952decfca7da31f4a85e41fe3489c1312f850b0626850de33c43a65 Oct 30 22:30:01.692567 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.692537 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a06b9e5_7af8_4214_8e6a_888ea52d2290.slice/crio-f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8 WatchSource:0}: Error finding container f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8: Status 404 returned error can't find the container with id f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8 Oct 30 22:30:01.694025 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:01.694004 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e8ffa0_136a_4eef_8d54_ad9e0ff89167.slice/crio-6357af2aeebc412d3250c307356ac1969fe29a1b7e70628a1f5bb460be99fcbc WatchSource:0}: Error finding container 6357af2aeebc412d3250c307356ac1969fe29a1b7e70628a1f5bb460be99fcbc: Status 404 returned error can't find the container with id 6357af2aeebc412d3250c307356ac1969fe29a1b7e70628a1f5bb460be99fcbc Oct 30 22:30:01.774710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.774667 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerStarted","Data":"b28770c6dcb92f2442dc1e2f6d6b8e82cd4a748ee35c96137ad50c59055f34c0"} Oct 30 22:30:01.776063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.776036 4292 generic.go:356] "Generic (PLEG): container finished" podID="b3e077c1-045f-4768-9778-ec5f714b6183" containerID="0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c" exitCode=0 Oct 30 22:30:01.776296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.776091 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" Oct 30 22:30:01.776296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.776099 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" event={"ID":"b3e077c1-045f-4768-9778-ec5f714b6183","Type":"ContainerDied","Data":"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c"} Oct 30 22:30:01.776296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.776136 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb" event={"ID":"b3e077c1-045f-4768-9778-ec5f714b6183","Type":"ContainerDied","Data":"23dd76fa88441e3c8bd7fe9ed8ef664efd4bea7c7ced8f19861d191402eef983"} Oct 30 22:30:01.776296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.776152 4292 scope.go:122] "RemoveContainer" containerID="0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c" Oct 30 22:30:01.777787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.777762 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerStarted","Data":"92d5b53d60ead7108a08763f6e199e26a1417b37665ef190f709cc65d6b82608"} Oct 30 22:30:01.779236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.779216 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerStarted","Data":"ba16d485690123072f31bb4269ba9f8586c6c1092517e19be77ca80d3393b8e6"} Oct 30 22:30:01.780765 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.780743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerStarted","Data":"7e4050067ca7bd0b4b4bd1791493609c2f5c4a3a293e39f374cfad98b0431d1f"} Oct 30 22:30:01.782273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.782254 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerStarted","Data":"feb42c16f764763767154dfcd1e40c27c65426d187c4ed83dee0feae9b96eb88"} Oct 30 22:30:01.783618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.783593 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerStarted","Data":"afddd8c6e7aa5d94df8e88eb6b5d0fb3ada922ec70e40a40f2026298f9ab02c5"} Oct 30 22:30:01.784810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.784788 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerStarted","Data":"541ea9759952decfca7da31f4a85e41fe3489c1312f850b0626850de33c43a65"} Oct 30 22:30:01.787760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.787686 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerStarted","Data":"f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8"} Oct 30 22:30:01.790147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.789197 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerStarted","Data":"1f092b1660692d13075785a86874451d5ca638bb8f6005112e8ea7e90dc72400"} Oct 30 22:30:01.816236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.816203 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerStarted","Data":"1bf472415f2be922f8d7ac63f4df31658a4aaf000cf8576a8f87d3ebaedca8ba"} Oct 30 22:30:01.818482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.818449 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerStarted","Data":"447b841418c7d0b9ffe21d18fa7a04c989c2a43a0c9d5c7ac1c3d23525d7707f"} Oct 30 22:30:01.818628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.818508 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerStarted","Data":"99d275a41b81b690d63d5d98443040070230e35593cb8dc21711d477cbb69b1a"} Oct 30 22:30:01.820243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.820212 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerStarted","Data":"6357af2aeebc412d3250c307356ac1969fe29a1b7e70628a1f5bb460be99fcbc"} Oct 30 22:30:01.820555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.820531 4292 scope.go:122] "RemoveContainer" containerID="0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c" Oct 30 22:30:01.820937 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:01.820906 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c\": container with ID starting with 0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c not found: ID does not exist" containerID="0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c" Oct 30 22:30:01.821052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:01.820947 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c"} err="failed to get container status \"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c\": rpc error: code = NotFound desc = could not find container \"0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c\": container with ID starting with 0ae05a206ba4c16dc5543189a7b5fc1bdba631a58b50ac58301edcd0e0d8259c not found: ID does not exist" Oct 30 22:30:02.020146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.020060 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:30:02.027050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.027023 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8e1227-57789d58ff-b6gbr"] Oct 30 22:30:02.151354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.151321 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" path="/var/lib/kubelet/pods/d6441d57-ca7e-4f63-9dc5-6e0088bd81c0/volumes" Oct 30 22:30:02.151688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.151673 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:30:02.151735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.151704 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8e1227-849c55f79-5s2x8"] Oct 30 22:30:02.254638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.254592 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k"] Oct 30 22:30:02.255375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255358 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" containerName="payload-pre-processing" Oct 30 22:30:02.255434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255377 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" containerName="payload-pre-processing" Oct 30 22:30:02.255434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255401 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" Oct 30 22:30:02.255434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255406 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" Oct 30 22:30:02.255532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255520 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" containerName="main" Oct 30 22:30:02.255568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.255538 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6441d57-ca7e-4f63-9dc5-6e0088bd81c0" containerName="payload-pre-processing" Oct 30 22:30:02.259202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.259186 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.313838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.313807 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:02.319900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.319506 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k"] Oct 30 22:30:02.409770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.409739 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.409961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.409834 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.409961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.409892 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t42w\" (UniqueName: \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.501839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.501806 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:30:02.506930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.506896 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-963c3a/premium-w3-963c3a-kserve-76999d4f88-7hdxb"] Oct 30 22:30:02.511804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.511778 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.511904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.511817 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.511904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.511846 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t42w\" (UniqueName: \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.512697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.512672 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.514252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.514228 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.579611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.579563 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t42w\" (UniqueName: \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\") pod \"maas-api-revoke-sub-a2aa98a1-6k65k\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.640083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.640025 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" podStartSLOduration=2.640010426 podStartE2EDuration="2.640010426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.636758783 +0000 UTC m=+2130.605578346" watchObservedRunningTime="2026-10-30 22:30:02.640010426 +0000 UTC m=+2130.608829989" Oct 30 22:30:02.830972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.830868 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerStarted","Data":"9793af5d5fc2f5cfce097600103dba79b0c600152072a5f1dd12601242257a31"} Oct 30 22:30:02.832710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.832669 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerStarted","Data":"ed6840d35255567d24758993472f090819821a81544963eecfe24578aba858c7"} Oct 30 22:30:02.834188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.834162 4292 generic.go:356] "Generic (PLEG): container finished" podID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerID="2c6f3bcc46fab35a7fea4cd9e6f04154a2449a968e62d75568c5d767740eb6dd" exitCode=60 Oct 30 22:30:02.834295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.834236 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerDied","Data":"2c6f3bcc46fab35a7fea4cd9e6f04154a2449a968e62d75568c5d767740eb6dd"} Oct 30 22:30:02.834483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.834466 4292 scope.go:122] "RemoveContainer" containerID="2c6f3bcc46fab35a7fea4cd9e6f04154a2449a968e62d75568c5d767740eb6dd" Oct 30 22:30:02.835887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.835864 4292 generic.go:356] "Generic (PLEG): container finished" podID="08061f89-9348-4dca-ba8f-78d7447a875b" containerID="8196730c7089ab2b6451c0333813dd23849de5335648c8019390748418d8ed5d" exitCode=60 Oct 30 22:30:02.835974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.835931 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerDied","Data":"8196730c7089ab2b6451c0333813dd23849de5335648c8019390748418d8ed5d"} Oct 30 22:30:02.836146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.836131 4292 scope.go:122] "RemoveContainer" containerID="8196730c7089ab2b6451c0333813dd23849de5335648c8019390748418d8ed5d" Oct 30 22:30:02.837949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.837734 4292 generic.go:356] "Generic (PLEG): container finished" podID="530df160-469a-41ee-861c-c1e556fa2016" containerID="447b841418c7d0b9ffe21d18fa7a04c989c2a43a0c9d5c7ac1c3d23525d7707f" exitCode=60 Oct 30 22:30:02.838079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.838063 4292 scope.go:122] "RemoveContainer" containerID="447b841418c7d0b9ffe21d18fa7a04c989c2a43a0c9d5c7ac1c3d23525d7707f" Oct 30 22:30:02.838455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.838394 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerDied","Data":"447b841418c7d0b9ffe21d18fa7a04c989c2a43a0c9d5c7ac1c3d23525d7707f"} Oct 30 22:30:02.840646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.840623 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerStarted","Data":"69f405c9017a96228a3c82e277a0e2ad6acd122dc2e2a65f4c187c92499d82e8"} Oct 30 22:30:02.842362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.842344 4292 generic.go:356] "Generic (PLEG): container finished" podID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerID="dbd4462eb04eb5e82dbf670829734d54b7984c299fea97626105c3605821c9d1" exitCode=60 Oct 30 22:30:02.842438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.842421 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerDied","Data":"dbd4462eb04eb5e82dbf670829734d54b7984c299fea97626105c3605821c9d1"} Oct 30 22:30:02.842664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.842647 4292 scope.go:122] "RemoveContainer" containerID="dbd4462eb04eb5e82dbf670829734d54b7984c299fea97626105c3605821c9d1" Oct 30 22:30:02.845306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.845284 4292 generic.go:356] "Generic (PLEG): container finished" podID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerID="06aedd94f83986cfab6f4b4a7035d5c071c7714c3ddea4a46766f236932b0c1a" exitCode=60 Oct 30 22:30:02.845469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.845355 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerDied","Data":"06aedd94f83986cfab6f4b4a7035d5c071c7714c3ddea4a46766f236932b0c1a"} Oct 30 22:30:02.845791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.845700 4292 scope.go:122] "RemoveContainer" containerID="06aedd94f83986cfab6f4b4a7035d5c071c7714c3ddea4a46766f236932b0c1a" Oct 30 22:30:02.848048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.848020 4292 generic.go:356] "Generic (PLEG): container finished" podID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerID="633c144a0ca3b8c96c53a73f5b94c3576aaf2c809f4a6c82dfbd4c070c3556f3" exitCode=60 Oct 30 22:30:02.848160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.848115 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerDied","Data":"633c144a0ca3b8c96c53a73f5b94c3576aaf2c809f4a6c82dfbd4c070c3556f3"} Oct 30 22:30:02.848465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.848340 4292 scope.go:122] "RemoveContainer" containerID="633c144a0ca3b8c96c53a73f5b94c3576aaf2c809f4a6c82dfbd4c070c3556f3" Oct 30 22:30:02.849855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.849835 4292 generic.go:356] "Generic (PLEG): container finished" podID="42b99f5a-0442-454d-9e8e-a61459da0505" containerID="088c402d9bed29e61c4d3134a9610952a2b8c1edc50a9c22a4d91afda24b9667" exitCode=60 Oct 30 22:30:02.849938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.849914 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerDied","Data":"088c402d9bed29e61c4d3134a9610952a2b8c1edc50a9c22a4d91afda24b9667"} Oct 30 22:30:02.850113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.850096 4292 scope.go:122] "RemoveContainer" containerID="088c402d9bed29e61c4d3134a9610952a2b8c1edc50a9c22a4d91afda24b9667" Oct 30 22:30:02.852059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.851565 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerStarted","Data":"3105a1ff9d6ed7549036323b8243a56edc945348e480088aeb28b552a17e88df"} Oct 30 22:30:02.853200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.853181 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerStarted","Data":"b0b0d99ec60396eabb6fb7cfbe0c660b135d2f9ee9046014d18f55816d750d30"} Oct 30 22:30:02.877177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.877144 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:02.949893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:02.949846 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" podStartSLOduration=2.9498317849999998 podStartE2EDuration="2.949831785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.94922199 +0000 UTC m=+2130.918041550" watchObservedRunningTime="2026-10-30 22:30:02.949831785 +0000 UTC m=+2130.918651348" Oct 30 22:30:03.189915 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.189864 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" podStartSLOduration=3.189849506 podStartE2EDuration="3.189849506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.188645353 +0000 UTC m=+2131.157464918" watchObservedRunningTime="2026-10-30 22:30:03.189849506 +0000 UTC m=+2131.158669067" Oct 30 22:30:03.293510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.293289 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:03.293510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.293454 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:03.295454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.295435 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k"] Oct 30 22:30:03.427601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.427541 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" podStartSLOduration=3.427528107 podStartE2EDuration="3.427528107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.425851273 +0000 UTC m=+2131.394670837" watchObservedRunningTime="2026-10-30 22:30:03.427528107 +0000 UTC m=+2131.396347671" Oct 30 22:30:03.862768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.862735 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerStarted","Data":"8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4"} Oct 30 22:30:03.864053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.864028 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" event={"ID":"09fdea95-a4da-4d8f-a70e-504b5202bc0a","Type":"ContainerStarted","Data":"a523601c8a78ee076868ae497bbfb51d8e07c964646847b6d616e1b46efe9561"} Oct 30 22:30:03.865465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.865441 4292 generic.go:356] "Generic (PLEG): container finished" podID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerID="3105a1ff9d6ed7549036323b8243a56edc945348e480088aeb28b552a17e88df" exitCode=60 Oct 30 22:30:03.865550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.865519 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerDied","Data":"3105a1ff9d6ed7549036323b8243a56edc945348e480088aeb28b552a17e88df"} Oct 30 22:30:03.865777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.865759 4292 scope.go:122] "RemoveContainer" containerID="3105a1ff9d6ed7549036323b8243a56edc945348e480088aeb28b552a17e88df" Oct 30 22:30:03.867200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.867173 4292 generic.go:356] "Generic (PLEG): container finished" podID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerID="b0b0d99ec60396eabb6fb7cfbe0c660b135d2f9ee9046014d18f55816d750d30" exitCode=60 Oct 30 22:30:03.867293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.867241 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerDied","Data":"b0b0d99ec60396eabb6fb7cfbe0c660b135d2f9ee9046014d18f55816d750d30"} Oct 30 22:30:03.867474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.867454 4292 scope.go:122] "RemoveContainer" containerID="b0b0d99ec60396eabb6fb7cfbe0c660b135d2f9ee9046014d18f55816d750d30" Oct 30 22:30:03.869127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.869106 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerID="9793af5d5fc2f5cfce097600103dba79b0c600152072a5f1dd12601242257a31" exitCode=60 Oct 30 22:30:03.869289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.869186 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerDied","Data":"9793af5d5fc2f5cfce097600103dba79b0c600152072a5f1dd12601242257a31"} Oct 30 22:30:03.869406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.869387 4292 scope.go:122] "RemoveContainer" containerID="9793af5d5fc2f5cfce097600103dba79b0c600152072a5f1dd12601242257a31" Oct 30 22:30:03.870875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.870834 4292 generic.go:356] "Generic (PLEG): container finished" podID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerID="ed6840d35255567d24758993472f090819821a81544963eecfe24578aba858c7" exitCode=60 Oct 30 22:30:03.870917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.870869 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerDied","Data":"ed6840d35255567d24758993472f090819821a81544963eecfe24578aba858c7"} Oct 30 22:30:03.871171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.871150 4292 scope.go:122] "RemoveContainer" containerID="ed6840d35255567d24758993472f090819821a81544963eecfe24578aba858c7" Oct 30 22:30:03.872882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.872862 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerStarted","Data":"41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25"} Oct 30 22:30:03.874727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.874703 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerStarted","Data":"6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab"} Oct 30 22:30:03.876528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.876506 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerStarted","Data":"0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778"} Oct 30 22:30:03.878166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.878144 4292 generic.go:356] "Generic (PLEG): container finished" podID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerID="69f405c9017a96228a3c82e277a0e2ad6acd122dc2e2a65f4c187c92499d82e8" exitCode=60 Oct 30 22:30:03.878304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.878283 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerDied","Data":"69f405c9017a96228a3c82e277a0e2ad6acd122dc2e2a65f4c187c92499d82e8"} Oct 30 22:30:03.878637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.878616 4292 scope.go:122] "RemoveContainer" containerID="69f405c9017a96228a3c82e277a0e2ad6acd122dc2e2a65f4c187c92499d82e8" Oct 30 22:30:03.880582 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.880541 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerStarted","Data":"c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6"} Oct 30 22:30:03.882411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.882391 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerStarted","Data":"602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c"} Oct 30 22:30:03.910056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:03.910016 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" podStartSLOduration=3.910002466 podStartE2EDuration="3.910002466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.908842488 +0000 UTC m=+2131.877662051" watchObservedRunningTime="2026-10-30 22:30:03.910002466 +0000 UTC m=+2131.878822030" Oct 30 22:30:04.149204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.149175 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10882900-cceb-48dd-bf40-486a132bed75" path="/var/lib/kubelet/pods/10882900-cceb-48dd-bf40-486a132bed75/volumes" Oct 30 22:30:04.149677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.149660 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e077c1-045f-4768-9778-ec5f714b6183" path="/var/lib/kubelet/pods/b3e077c1-045f-4768-9778-ec5f714b6183/volumes" Oct 30 22:30:04.266688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.266602 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" podStartSLOduration=4.266590584 podStartE2EDuration="4.266590584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.265277829 +0000 UTC m=+2132.234097395" watchObservedRunningTime="2026-10-30 22:30:04.266590584 +0000 UTC m=+2132.235410147" Oct 30 22:30:04.389853 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.389792 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" podStartSLOduration=4.38977266 podStartE2EDuration="4.38977266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.383751131 +0000 UTC m=+2132.352570695" watchObservedRunningTime="2026-10-30 22:30:04.38977266 +0000 UTC m=+2132.358592226" Oct 30 22:30:04.623244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.623201 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" podStartSLOduration=4.623187923 podStartE2EDuration="4.623187923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.620723953 +0000 UTC m=+2132.589543520" watchObservedRunningTime="2026-10-30 22:30:04.623187923 +0000 UTC m=+2132.592007487" Oct 30 22:30:04.876380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.876320 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" podStartSLOduration=4.876307839 podStartE2EDuration="4.876307839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.872081517 +0000 UTC m=+2132.840901093" watchObservedRunningTime="2026-10-30 22:30:04.876307839 +0000 UTC m=+2132.845127402" Oct 30 22:30:04.896039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.896005 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerStarted","Data":"b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5"} Oct 30 22:30:04.897993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.897964 4292 generic.go:356] "Generic (PLEG): container finished" podID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerID="c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6" exitCode=60 Oct 30 22:30:04.898134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.898041 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerDied","Data":"c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6"} Oct 30 22:30:04.898134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.898097 4292 scope.go:122] "RemoveContainer" containerID="dbd4462eb04eb5e82dbf670829734d54b7984c299fea97626105c3605821c9d1" Oct 30 22:30:04.898290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.898272 4292 scope.go:122] "RemoveContainer" containerID="c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6" Oct 30 22:30:04.898641 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.898620 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm_odh-ai-gateway-infra(28edfe07-b99a-43e5-b724-ac304d6de0ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" Oct 30 22:30:04.900048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.900024 4292 generic.go:356] "Generic (PLEG): container finished" podID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerID="4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71" exitCode=60 Oct 30 22:30:04.900152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.900091 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerDied","Data":"4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71"} Oct 30 22:30:04.900297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.900283 4292 scope.go:122] "RemoveContainer" containerID="4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71" Oct 30 22:30:04.900504 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.900484 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v_odh-ai-gateway-infra(0f8e9159-6c6e-4259-891c-d292138bf8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" Oct 30 22:30:04.901956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.901934 4292 generic.go:356] "Generic (PLEG): container finished" podID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerID="602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c" exitCode=60 Oct 30 22:30:04.902025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.902007 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerDied","Data":"602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c"} Oct 30 22:30:04.902250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.902229 4292 scope.go:122] "RemoveContainer" containerID="602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c" Oct 30 22:30:04.902448 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.902429 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9_odh-ai-gateway-infra(4a67acb1-27b6-422c-9ba0-c6663b09fa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" Oct 30 22:30:04.904088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.904063 4292 generic.go:356] "Generic (PLEG): container finished" podID="42b99f5a-0442-454d-9e8e-a61459da0505" containerID="8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4" exitCode=60 Oct 30 22:30:04.904208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.904105 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerDied","Data":"8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4"} Oct 30 22:30:04.905046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.905020 4292 scope.go:122] "RemoveContainer" containerID="8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4" Oct 30 22:30:04.905544 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.905516 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-x4k9n_odh-ai-gateway-infra(42b99f5a-0442-454d-9e8e-a61459da0505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" Oct 30 22:30:04.919250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.919222 4292 generic.go:356] "Generic (PLEG): container finished" podID="09fdea95-a4da-4d8f-a70e-504b5202bc0a" containerID="c1174c6d939b393e8673558f66ed7db2b46724aea4d4d90dc8faaa83fc840907" exitCode=0 Oct 30 22:30:04.919372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.919292 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" event={"ID":"09fdea95-a4da-4d8f-a70e-504b5202bc0a","Type":"ContainerDied","Data":"c1174c6d939b393e8673558f66ed7db2b46724aea4d4d90dc8faaa83fc840907"} Oct 30 22:30:04.922009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.921864 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerStarted","Data":"63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194"} Oct 30 22:30:04.924215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.924194 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerStarted","Data":"a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b"} Oct 30 22:30:04.925281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.925255 4292 scope.go:122] "RemoveContainer" containerID="06aedd94f83986cfab6f4b4a7035d5c071c7714c3ddea4a46766f236932b0c1a" Oct 30 22:30:04.926408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.926383 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerStarted","Data":"aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175"} Oct 30 22:30:04.928682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.928661 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerStarted","Data":"78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0"} Oct 30 22:30:04.930475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.930454 4292 generic.go:356] "Generic (PLEG): container finished" podID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerID="41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25" exitCode=60 Oct 30 22:30:04.930569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.930480 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerDied","Data":"41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25"} Oct 30 22:30:04.930723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.930709 4292 scope.go:122] "RemoveContainer" containerID="41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25" Oct 30 22:30:04.930974 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.930949 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp_odh-ai-gateway-infra(e12303c3-00b0-4d84-b007-a3f040b44ea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" Oct 30 22:30:04.932461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.932437 4292 generic.go:356] "Generic (PLEG): container finished" podID="08061f89-9348-4dca-ba8f-78d7447a875b" containerID="6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab" exitCode=60 Oct 30 22:30:04.932553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.932518 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerDied","Data":"6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab"} Oct 30 22:30:04.932719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.932704 4292 scope.go:122] "RemoveContainer" containerID="6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab" Oct 30 22:30:04.932940 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.932917 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h_odh-ai-gateway-infra(08061f89-9348-4dca-ba8f-78d7447a875b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" Oct 30 22:30:04.934459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.934438 4292 generic.go:356] "Generic (PLEG): container finished" podID="530df160-469a-41ee-861c-c1e556fa2016" containerID="0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778" exitCode=60 Oct 30 22:30:04.934559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.934489 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerDied","Data":"0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778"} Oct 30 22:30:04.934720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.934707 4292 scope.go:122] "RemoveContainer" containerID="0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778" Oct 30 22:30:04.934905 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:04.934890 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr_odh-ai-gateway-infra(530df160-469a-41ee-861c-c1e556fa2016)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" podUID="530df160-469a-41ee-861c-c1e556fa2016" Oct 30 22:30:04.939381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.939364 4292 scope.go:122] "RemoveContainer" containerID="633c144a0ca3b8c96c53a73f5b94c3576aaf2c809f4a6c82dfbd4c070c3556f3" Oct 30 22:30:04.952116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.952096 4292 scope.go:122] "RemoveContainer" containerID="088c402d9bed29e61c4d3134a9610952a2b8c1edc50a9c22a4d91afda24b9667" Oct 30 22:30:04.964835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.964816 4292 scope.go:122] "RemoveContainer" containerID="2c6f3bcc46fab35a7fea4cd9e6f04154a2449a968e62d75568c5d767740eb6dd" Oct 30 22:30:04.979988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:04.979967 4292 scope.go:122] "RemoveContainer" containerID="8196730c7089ab2b6451c0333813dd23849de5335648c8019390748418d8ed5d" Oct 30 22:30:05.229605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.229548 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" podStartSLOduration=5.229535581 podStartE2EDuration="5.229535581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:05.228826445 +0000 UTC m=+2133.197646020" watchObservedRunningTime="2026-10-30 22:30:05.229535581 +0000 UTC m=+2133.198355139" Oct 30 22:30:05.307004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.306982 4292 scope.go:122] "RemoveContainer" containerID="447b841418c7d0b9ffe21d18fa7a04c989c2a43a0c9d5c7ac1c3d23525d7707f" Oct 30 22:30:05.470606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.470498 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" podStartSLOduration=5.470485373 podStartE2EDuration="5.470485373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:05.468913435 +0000 UTC m=+2133.437732998" watchObservedRunningTime="2026-10-30 22:30:05.470485373 +0000 UTC m=+2133.439304937" Oct 30 22:30:05.946955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.946908 4292 generic.go:356] "Generic (PLEG): container finished" podID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerID="b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5" exitCode=60 Oct 30 22:30:05.947293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.947268 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerDied","Data":"b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5"} Oct 30 22:30:05.947464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.947451 4292 scope.go:122] "RemoveContainer" containerID="69f405c9017a96228a3c82e277a0e2ad6acd122dc2e2a65f4c187c92499d82e8" Oct 30 22:30:05.948028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.948006 4292 scope.go:122] "RemoveContainer" containerID="b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5" Oct 30 22:30:05.948613 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.948563 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc_odh-ai-gateway-infra(d7e8ffa0-136a-4eef-8d54-ad9e0ff89167)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" Oct 30 22:30:05.953698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.953679 4292 scope.go:122] "RemoveContainer" containerID="c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6" Oct 30 22:30:05.954009 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.953985 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm_odh-ai-gateway-infra(28edfe07-b99a-43e5-b724-ac304d6de0ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" Oct 30 22:30:05.955462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.955442 4292 scope.go:122] "RemoveContainer" containerID="4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71" Oct 30 22:30:05.955897 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.955874 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v_odh-ai-gateway-infra(0f8e9159-6c6e-4259-891c-d292138bf8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" Oct 30 22:30:05.957420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.957393 4292 scope.go:122] "RemoveContainer" containerID="602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c" Oct 30 22:30:05.957719 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.957693 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9_odh-ai-gateway-infra(4a67acb1-27b6-422c-9ba0-c6663b09fa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" Oct 30 22:30:05.959472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.959449 4292 scope.go:122] "RemoveContainer" containerID="8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4" Oct 30 22:30:05.959801 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.959762 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-x4k9n_odh-ai-gateway-infra(42b99f5a-0442-454d-9e8e-a61459da0505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" Oct 30 22:30:05.961261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.961240 4292 generic.go:356] "Generic (PLEG): container finished" podID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerID="63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194" exitCode=60 Oct 30 22:30:05.961354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.961268 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerDied","Data":"63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194"} Oct 30 22:30:05.961741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.961718 4292 scope.go:122] "RemoveContainer" containerID="63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194" Oct 30 22:30:05.962102 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.962070 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks_odh-ai-gateway-infra(34d2a0e3-b6dc-417f-89da-4fd53c19b981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" Oct 30 22:30:05.963318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.963299 4292 generic.go:356] "Generic (PLEG): container finished" podID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerID="a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b" exitCode=60 Oct 30 22:30:05.963452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.963371 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerDied","Data":"a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b"} Oct 30 22:30:05.963786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.963762 4292 scope.go:122] "RemoveContainer" containerID="a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b" Oct 30 22:30:05.964040 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.964012 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6_odh-ai-gateway-infra(d2ae0e55-b704-4446-a5f8-855033b855f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" Oct 30 22:30:05.965551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.965531 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerID="aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175" exitCode=60 Oct 30 22:30:05.965663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.965630 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerDied","Data":"aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175"} Oct 30 22:30:05.965816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.965798 4292 scope.go:122] "RemoveContainer" containerID="aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175" Oct 30 22:30:05.966008 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.965990 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh_odh-ai-gateway-infra(9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" Oct 30 22:30:05.966783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.966761 4292 scope.go:122] "RemoveContainer" containerID="3105a1ff9d6ed7549036323b8243a56edc945348e480088aeb28b552a17e88df" Oct 30 22:30:05.968120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.968097 4292 generic.go:356] "Generic (PLEG): container finished" podID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerID="78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0" exitCode=60 Oct 30 22:30:05.968216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.968192 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerDied","Data":"78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0"} Oct 30 22:30:05.968422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.968408 4292 scope.go:122] "RemoveContainer" containerID="78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0" Oct 30 22:30:05.968650 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.968627 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm_odh-ai-gateway-infra(9a06b9e5-7af8-4214-8e6a-888ea52d2290)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" Oct 30 22:30:05.970088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.970067 4292 scope.go:122] "RemoveContainer" containerID="41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25" Oct 30 22:30:05.970278 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.970261 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp_odh-ai-gateway-infra(e12303c3-00b0-4d84-b007-a3f040b44ea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" Oct 30 22:30:05.971886 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.971867 4292 scope.go:122] "RemoveContainer" containerID="6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab" Oct 30 22:30:05.972125 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.972101 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h_odh-ai-gateway-infra(08061f89-9348-4dca-ba8f-78d7447a875b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" Oct 30 22:30:05.973715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.973686 4292 scope.go:122] "RemoveContainer" containerID="0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778" Oct 30 22:30:05.973899 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:05.973876 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr_odh-ai-gateway-infra(530df160-469a-41ee-861c-c1e556fa2016)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" podUID="530df160-469a-41ee-861c-c1e556fa2016" Oct 30 22:30:05.982696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.982678 4292 scope.go:122] "RemoveContainer" containerID="b0b0d99ec60396eabb6fb7cfbe0c660b135d2f9ee9046014d18f55816d750d30" Oct 30 22:30:05.995747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:05.995722 4292 scope.go:122] "RemoveContainer" containerID="9793af5d5fc2f5cfce097600103dba79b0c600152072a5f1dd12601242257a31" Oct 30 22:30:06.011445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.011421 4292 scope.go:122] "RemoveContainer" containerID="ed6840d35255567d24758993472f090819821a81544963eecfe24578aba858c7" Oct 30 22:30:06.139222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.139198 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:06.272076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.272008 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\") pod \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " Oct 30 22:30:06.272076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.272061 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\") pod \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " Oct 30 22:30:06.272243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.272127 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\" (UniqueName: \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\") pod \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\" (UID: \"09fdea95-a4da-4d8f-a70e-504b5202bc0a\") " Oct 30 22:30:06.272353 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.272325 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca" pod "09fdea95-a4da-4d8f-a70e-504b5202bc0a" (UID: "09fdea95-a4da-4d8f-a70e-504b5202bc0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:06.272645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.272619 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:06.274133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.274110 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca" pod "09fdea95-a4da-4d8f-a70e-504b5202bc0a" (UID: "09fdea95-a4da-4d8f-a70e-504b5202bc0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:06.274243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.274230 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w" pod "09fdea95-a4da-4d8f-a70e-504b5202bc0a" (UID: "09fdea95-a4da-4d8f-a70e-504b5202bc0a"). InnerVolumeSpecName "kube-api-access-6t42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:06.374339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.374314 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fdea95-a4da-4d8f-a70e-504b5202bc0a-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:06.374339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.374339 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t42w\" (UniqueName: \"kubernetes.io/projected/09fdea95-a4da-4d8f-a70e-504b5202bc0a-kube-api-access-6t42w\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:06.982688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.982659 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" Oct 30 22:30:06.982852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.982666 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k" event={"ID":"09fdea95-a4da-4d8f-a70e-504b5202bc0a","Type":"ContainerDied","Data":"a523601c8a78ee076868ae497bbfb51d8e07c964646847b6d616e1b46efe9561"} Oct 30 22:30:06.982852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.982785 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a523601c8a78ee076868ae497bbfb51d8e07c964646847b6d616e1b46efe9561" Oct 30 22:30:06.984769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.984750 4292 scope.go:122] "RemoveContainer" containerID="63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194" Oct 30 22:30:06.985036 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:06.985017 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks_odh-ai-gateway-infra(34d2a0e3-b6dc-417f-89da-4fd53c19b981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" Oct 30 22:30:06.986381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.986358 4292 scope.go:122] "RemoveContainer" containerID="a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b" Oct 30 22:30:06.986682 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:06.986658 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6_odh-ai-gateway-infra(d2ae0e55-b704-4446-a5f8-855033b855f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" Oct 30 22:30:06.988058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.988041 4292 scope.go:122] "RemoveContainer" containerID="aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175" Oct 30 22:30:06.988270 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:06.988247 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh_odh-ai-gateway-infra(9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" Oct 30 22:30:06.989616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.989598 4292 scope.go:122] "RemoveContainer" containerID="78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0" Oct 30 22:30:06.989843 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:06.989816 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm_odh-ai-gateway-infra(9a06b9e5-7af8-4214-8e6a-888ea52d2290)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" Oct 30 22:30:06.991302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:06.991286 4292 scope.go:122] "RemoveContainer" containerID="b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5" Oct 30 22:30:06.991484 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:06.991467 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc_odh-ai-gateway-infra(d7e8ffa0-136a-4eef-8d54-ad9e0ff89167)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" Oct 30 22:30:11.531988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:11.531946 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:30:11.643395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:11.643359 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:30:13.315974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:13.315515 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:13.316608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:13.316555 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:15.140032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:15.140000 4292 scope.go:122] "RemoveContainer" containerID="6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab" Oct 30 22:30:15.140496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:15.140271 4292 scope.go:122] "RemoveContainer" containerID="602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c" Oct 30 22:30:16.078535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.078505 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerStarted","Data":"642da291bbf88ec014d179576fc5301647569737846de8c65d040aebda2b110a"} Oct 30 22:30:16.080488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.080462 4292 generic.go:356] "Generic (PLEG): container finished" podID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerID="267d245cc5fb9efe23f93241117d60b2bfccbfd5919c7c3ae88d33261f9ed271" exitCode=60 Oct 30 22:30:16.080612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.080535 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerDied","Data":"267d245cc5fb9efe23f93241117d60b2bfccbfd5919c7c3ae88d33261f9ed271"} Oct 30 22:30:16.080612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.080600 4292 scope.go:122] "RemoveContainer" containerID="602a207909d60c5a6ecec9083a269cda516eb71bec1353f73129c2152ff1d89c" Oct 30 22:30:16.080897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.080881 4292 scope.go:122] "RemoveContainer" containerID="267d245cc5fb9efe23f93241117d60b2bfccbfd5919c7c3ae88d33261f9ed271" Oct 30 22:30:16.081235 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:16.081202 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9_odh-ai-gateway-infra(4a67acb1-27b6-422c-9ba0-c6663b09fa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" Oct 30 22:30:16.139584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.139544 4292 scope.go:122] "RemoveContainer" containerID="b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5" Oct 30 22:30:16.139707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.139687 4292 scope.go:122] "RemoveContainer" containerID="8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4" Oct 30 22:30:16.139816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.139798 4292 scope.go:122] "RemoveContainer" containerID="c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6" Oct 30 22:30:16.139917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.139899 4292 scope.go:122] "RemoveContainer" containerID="a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b" Oct 30 22:30:16.532755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.532708 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" podUID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" containerName="payload-processing" containerID="cri-o://e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775" gracePeriod=25 Oct 30 22:30:16.644174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.644127 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" podUID="e217501d-0452-4108-8821-5ecb214095ee" containerName="payload-pre-processing" containerID="cri-o://b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76" gracePeriod=25 Oct 30 22:30:16.863446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.863416 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:30:16.878514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.878476 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:30:16.905418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.905396 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:30:16.912013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.911991 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\" (UniqueName: \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\") pod \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " Oct 30 22:30:16.912167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.912155 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\") pod \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\" (UID: \"be72a21d-df2c-4cbe-90e4-cfc491e7788c\") " Oct 30 22:30:16.912491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.912469 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume" pod "be72a21d-df2c-4cbe-90e4-cfc491e7788c" (UID: "be72a21d-df2c-4cbe-90e4-cfc491e7788c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:16.912742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.912724 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72a21d-df2c-4cbe-90e4-cfc491e7788c-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.913991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.913972 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r" pod "be72a21d-df2c-4cbe-90e4-cfc491e7788c" (UID: "be72a21d-df2c-4cbe-90e4-cfc491e7788c"). InnerVolumeSpecName "kube-api-access-tvn8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:16.976988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:16.976956 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:30:17.013857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.013830 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\") pod \"e217501d-0452-4108-8821-5ecb214095ee\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " Oct 30 22:30:17.014074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.014052 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\" (UniqueName: \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\") pod \"e217501d-0452-4108-8821-5ecb214095ee\" (UID: \"e217501d-0452-4108-8821-5ecb214095ee\") " Oct 30 22:30:17.014178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.014161 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume" pod "e217501d-0452-4108-8821-5ecb214095ee" (UID: "e217501d-0452-4108-8821-5ecb214095ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.014553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.014533 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e217501d-0452-4108-8821-5ecb214095ee-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.014553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.014553 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvn8r\" (UniqueName: \"kubernetes.io/projected/be72a21d-df2c-4cbe-90e4-cfc491e7788c-kube-api-access-tvn8r\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.016022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.016002 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9" pod "e217501d-0452-4108-8821-5ecb214095ee" (UID: "e217501d-0452-4108-8821-5ecb214095ee"). InnerVolumeSpecName "kube-api-access-g92d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.096050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.095968 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerStarted","Data":"849c2501602daed8e5d120c87a39433c8d7558ad73915cd93b548d0b3a2ceeeb"} Oct 30 22:30:17.097986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.097962 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerStarted","Data":"7e81fbc01197454142508f8c2734ad93c263c8b7fd0c6fc8929b7961d0b18a97"} Oct 30 22:30:17.099832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.099809 4292 generic.go:356] "Generic (PLEG): container finished" podID="08061f89-9348-4dca-ba8f-78d7447a875b" containerID="642da291bbf88ec014d179576fc5301647569737846de8c65d040aebda2b110a" exitCode=60 Oct 30 22:30:17.099942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.099882 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerDied","Data":"642da291bbf88ec014d179576fc5301647569737846de8c65d040aebda2b110a"} Oct 30 22:30:17.100031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.099938 4292 scope.go:122] "RemoveContainer" containerID="6222fbac5d1ab22d3d1a707da7de95f48b6e94838dacf575cae59e80d1fd4cab" Oct 30 22:30:17.100230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.100215 4292 scope.go:122] "RemoveContainer" containerID="642da291bbf88ec014d179576fc5301647569737846de8c65d040aebda2b110a" Oct 30 22:30:17.100497 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:17.100469 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h_odh-ai-gateway-infra(08061f89-9348-4dca-ba8f-78d7447a875b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" Oct 30 22:30:17.102031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.102010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerStarted","Data":"da68b32737948d64046bc314b06be8a235781d8ed0d40d653ffa7f69734e7323"} Oct 30 22:30:17.103293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.103271 4292 generic.go:356] "Generic (PLEG): container finished" podID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" containerID="e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775" exitCode=0 Oct 30 22:30:17.103378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.103324 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" Oct 30 22:30:17.103378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.103348 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" event={"ID":"be72a21d-df2c-4cbe-90e4-cfc491e7788c","Type":"ContainerDied","Data":"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775"} Oct 30 22:30:17.103477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.103393 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr" event={"ID":"be72a21d-df2c-4cbe-90e4-cfc491e7788c","Type":"ContainerDied","Data":"9aa89558e3bc57e064e48707e03811480cb878823143a122053e2264edf20ef7"} Oct 30 22:30:17.105276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.105256 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerStarted","Data":"1b441e880aa4c169d2748372f7dadb107f99d25f23e9a6f5df0305048f4c8dda"} Oct 30 22:30:17.108394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.108368 4292 generic.go:356] "Generic (PLEG): container finished" podID="e217501d-0452-4108-8821-5ecb214095ee" containerID="b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76" exitCode=0 Oct 30 22:30:17.108497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.108440 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" event={"ID":"e217501d-0452-4108-8821-5ecb214095ee","Type":"ContainerDied","Data":"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76"} Oct 30 22:30:17.108497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.108477 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" event={"ID":"e217501d-0452-4108-8821-5ecb214095ee","Type":"ContainerDied","Data":"fb01bbc277b692cc8674d4fe96c6fc742e64e75e7303a241f7e2073afe269d72"} Oct 30 22:30:17.108708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.108683 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54" Oct 30 22:30:17.115395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.115376 4292 scope.go:122] "RemoveContainer" containerID="e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775" Oct 30 22:30:17.116071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.116051 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g92d9\" (UniqueName: \"kubernetes.io/projected/e217501d-0452-4108-8821-5ecb214095ee-kube-api-access-g92d9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.125933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.125910 4292 scope.go:122] "RemoveContainer" containerID="e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775" Oct 30 22:30:17.126209 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:17.126186 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775\": container with ID starting with e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775 not found: ID does not exist" containerID="e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775" Oct 30 22:30:17.126279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.126220 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775"} err="failed to get container status \"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775\": rpc error: code = NotFound desc = could not find container \"e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775\": container with ID starting with e2aff51ed9e84485e4ec9b3d5558f8dacdcbeb992996b1921c5066f8f7c63775 not found: ID does not exist" Oct 30 22:30:17.126279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.126242 4292 scope.go:122] "RemoveContainer" containerID="b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76" Oct 30 22:30:17.135516 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.135496 4292 scope.go:122] "RemoveContainer" containerID="b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76" Oct 30 22:30:17.135776 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:17.135757 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76\": container with ID starting with b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76 not found: ID does not exist" containerID="b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76" Oct 30 22:30:17.135850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.135786 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76"} err="failed to get container status \"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76\": rpc error: code = NotFound desc = could not find container \"b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76\": container with ID starting with b4cb1f776947e45282d51078fe45a16e6e4d664d48b138dd6c89465e54946b76 not found: ID does not exist" Oct 30 22:30:17.139233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.139218 4292 scope.go:122] "RemoveContainer" containerID="41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25" Oct 30 22:30:17.214754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.214724 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:17.346344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.346200 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:17.746820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.746797 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:17.751431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.751406 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:30:17.828637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.828560 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:30:17.854114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854090 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\" (UniqueName: \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\") pod \"992b0a74-db4f-450e-a47c-1d420001bf95\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " Oct 30 22:30:17.854230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854220 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\") pod \"b9370076-ea62-4fcb-96f0-f8efd710e280\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " Oct 30 22:30:17.854297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854283 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\" (UniqueName: \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\") pod \"b9370076-ea62-4fcb-96f0-f8efd710e280\" (UID: \"b9370076-ea62-4fcb-96f0-f8efd710e280\") " Oct 30 22:30:17.854337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854319 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\" (UniqueName: \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\") pod \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " Oct 30 22:30:17.854398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854387 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\") pod \"992b0a74-db4f-450e-a47c-1d420001bf95\" (UID: \"992b0a74-db4f-450e-a47c-1d420001bf95\") " Oct 30 22:30:17.854530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854488 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\") pod \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\" (UID: \"4a67acb1-27b6-422c-9ba0-c6663b09fa46\") " Oct 30 22:30:17.854669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854649 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume" pod "b9370076-ea62-4fcb-96f0-f8efd710e280" (UID: "b9370076-ea62-4fcb-96f0-f8efd710e280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.854896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854867 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume" pod "992b0a74-db4f-450e-a47c-1d420001bf95" (UID: "992b0a74-db4f-450e-a47c-1d420001bf95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.854896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.854883 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca" pod "4a67acb1-27b6-422c-9ba0-c6663b09fa46" (UID: "4a67acb1-27b6-422c-9ba0-c6663b09fa46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.855321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.855296 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992b0a74-db4f-450e-a47c-1d420001bf95-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.855394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.855329 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a67acb1-27b6-422c-9ba0-c6663b09fa46-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.855394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.855347 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9370076-ea62-4fcb-96f0-f8efd710e280-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.856568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.856548 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn" pod "4a67acb1-27b6-422c-9ba0-c6663b09fa46" (UID: "4a67acb1-27b6-422c-9ba0-c6663b09fa46"). InnerVolumeSpecName "kube-api-access-9j5pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.856654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.856604 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4" pod "992b0a74-db4f-450e-a47c-1d420001bf95" (UID: "992b0a74-db4f-450e-a47c-1d420001bf95"). InnerVolumeSpecName "kube-api-access-pb4l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.856654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.856638 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8" pod "b9370076-ea62-4fcb-96f0-f8efd710e280" (UID: "b9370076-ea62-4fcb-96f0-f8efd710e280"). InnerVolumeSpecName "kube-api-access-fgsf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.957238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.957173 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgsf8\" (UniqueName: \"kubernetes.io/projected/b9370076-ea62-4fcb-96f0-f8efd710e280-kube-api-access-fgsf8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.957238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.957204 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5pn\" (UniqueName: \"kubernetes.io/projected/4a67acb1-27b6-422c-9ba0-c6663b09fa46-kube-api-access-9j5pn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.957238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:17.957217 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4l4\" (UniqueName: \"kubernetes.io/projected/992b0a74-db4f-450e-a47c-1d420001bf95-kube-api-access-pb4l4\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.002044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.002019 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:30:18.009819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.009796 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-56c628-78d789c84c-xbdxr"] Oct 30 22:30:18.124511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.124477 4292 generic.go:356] "Generic (PLEG): container finished" podID="992b0a74-db4f-450e-a47c-1d420001bf95" containerID="827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0" exitCode=0 Oct 30 22:30:18.124721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.124534 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" Oct 30 22:30:18.124721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.124603 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" event={"ID":"992b0a74-db4f-450e-a47c-1d420001bf95","Type":"ContainerDied","Data":"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0"} Oct 30 22:30:18.124721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.124668 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8" event={"ID":"992b0a74-db4f-450e-a47c-1d420001bf95","Type":"ContainerDied","Data":"3ce4b89584c36a16bcf23cf02c84b6fde290d6e2fb254f1dd1cbf6a15b70528e"} Oct 30 22:30:18.124721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.124694 4292 scope.go:122] "RemoveContainer" containerID="827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0" Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.126305 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.126299 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9" event={"ID":"4a67acb1-27b6-422c-9ba0-c6663b09fa46","Type":"ContainerDied","Data":"ba16d485690123072f31bb4269ba9f8586c6c1092517e19be77ca80d3393b8e6"} Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.129605 4292 generic.go:356] "Generic (PLEG): container finished" podID="42b99f5a-0442-454d-9e8e-a61459da0505" containerID="849c2501602daed8e5d120c87a39433c8d7558ad73915cd93b548d0b3a2ceeeb" exitCode=60 Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.129698 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerDied","Data":"849c2501602daed8e5d120c87a39433c8d7558ad73915cd93b548d0b3a2ceeeb"} Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.130080 4292 scope.go:122] "RemoveContainer" containerID="849c2501602daed8e5d120c87a39433c8d7558ad73915cd93b548d0b3a2ceeeb" Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.130346 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-x4k9n_odh-ai-gateway-infra(42b99f5a-0442-454d-9e8e-a61459da0505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.132059 4292 generic.go:356] "Generic (PLEG): container finished" podID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerID="7e81fbc01197454142508f8c2734ad93c263c8b7fd0c6fc8929b7961d0b18a97" exitCode=60 Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.132123 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerDied","Data":"7e81fbc01197454142508f8c2734ad93c263c8b7fd0c6fc8929b7961d0b18a97"} Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.132453 4292 scope.go:122] "RemoveContainer" containerID="7e81fbc01197454142508f8c2734ad93c263c8b7fd0c6fc8929b7961d0b18a97" Oct 30 22:30:18.133170 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.132697 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6_odh-ai-gateway-infra(d2ae0e55-b704-4446-a5f8-855033b855f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" Oct 30 22:30:18.134343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.134323 4292 generic.go:356] "Generic (PLEG): container finished" podID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerID="b8c57db269f35cdd21e8fad10cc5ab48405d2ff35bcf854ee62da79293e2ad2a" exitCode=60 Oct 30 22:30:18.134446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.134405 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerDied","Data":"b8c57db269f35cdd21e8fad10cc5ab48405d2ff35bcf854ee62da79293e2ad2a"} Oct 30 22:30:18.134821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.134802 4292 scope.go:122] "RemoveContainer" containerID="b8c57db269f35cdd21e8fad10cc5ab48405d2ff35bcf854ee62da79293e2ad2a" Oct 30 22:30:18.135079 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.135049 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp_odh-ai-gateway-infra(e12303c3-00b0-4d84-b007-a3f040b44ea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" Oct 30 22:30:18.137692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.137629 4292 scope.go:122] "RemoveContainer" containerID="827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0" Oct 30 22:30:18.138036 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.137910 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0\": container with ID starting with 827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0 not found: ID does not exist" containerID="827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0" Oct 30 22:30:18.138036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.137943 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0"} err="failed to get container status \"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0\": rpc error: code = NotFound desc = could not find container \"827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0\": container with ID starting with 827e6a6880f7105e7399bb76b142a89c013e3dad1545c7102940cc04d81eabf0 not found: ID does not exist" Oct 30 22:30:18.138036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.137964 4292 scope.go:122] "RemoveContainer" containerID="267d245cc5fb9efe23f93241117d60b2bfccbfd5919c7c3ae88d33261f9ed271" Oct 30 22:30:18.139293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.139275 4292 scope.go:122] "RemoveContainer" containerID="0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778" Oct 30 22:30:18.139455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.139438 4292 scope.go:122] "RemoveContainer" containerID="aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175" Oct 30 22:30:18.139964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.139589 4292 scope.go:122] "RemoveContainer" containerID="78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0" Oct 30 22:30:18.139964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.139739 4292 scope.go:122] "RemoveContainer" containerID="4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71" Oct 30 22:30:18.141800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.141768 4292 generic.go:356] "Generic (PLEG): container finished" podID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerID="da68b32737948d64046bc314b06be8a235781d8ed0d40d653ffa7f69734e7323" exitCode=60 Oct 30 22:30:18.149104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.146850 4292 generic.go:356] "Generic (PLEG): container finished" podID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerID="f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572" exitCode=0 Oct 30 22:30:18.149104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.146957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" Oct 30 22:30:18.150846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.150821 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" path="/var/lib/kubelet/pods/be72a21d-df2c-4cbe-90e4-cfc491e7788c/volumes" Oct 30 22:30:18.151292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.151272 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerDied","Data":"da68b32737948d64046bc314b06be8a235781d8ed0d40d653ffa7f69734e7323"} Oct 30 22:30:18.151365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.151312 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" event={"ID":"b9370076-ea62-4fcb-96f0-f8efd710e280","Type":"ContainerDied","Data":"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572"} Oct 30 22:30:18.151365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.151332 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp" event={"ID":"b9370076-ea62-4fcb-96f0-f8efd710e280","Type":"ContainerDied","Data":"9f04f854c3ae5b9670fc45144d61a3e34bce2bd939a608bb2477816d0c9c1890"} Oct 30 22:30:18.151595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.151558 4292 scope.go:122] "RemoveContainer" containerID="da68b32737948d64046bc314b06be8a235781d8ed0d40d653ffa7f69734e7323" Oct 30 22:30:18.151810 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.151791 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc_odh-ai-gateway-infra(d7e8ffa0-136a-4eef-8d54-ad9e0ff89167)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" Oct 30 22:30:18.153545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.153524 4292 generic.go:356] "Generic (PLEG): container finished" podID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerID="1b441e880aa4c169d2748372f7dadb107f99d25f23e9a6f5df0305048f4c8dda" exitCode=60 Oct 30 22:30:18.153782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.153554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerDied","Data":"1b441e880aa4c169d2748372f7dadb107f99d25f23e9a6f5df0305048f4c8dda"} Oct 30 22:30:18.154693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.153889 4292 scope.go:122] "RemoveContainer" containerID="1b441e880aa4c169d2748372f7dadb107f99d25f23e9a6f5df0305048f4c8dda" Oct 30 22:30:18.154693 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.154137 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm_odh-ai-gateway-infra(28edfe07-b99a-43e5-b724-ac304d6de0ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" Oct 30 22:30:18.165025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.165006 4292 scope.go:122] "RemoveContainer" containerID="8ca87ec1462ea64df96e3f9c6ce9c45f544745a4148a84399aaef8f8aa983fa4" Oct 30 22:30:18.308698 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.308669 4292 scope.go:122] "RemoveContainer" containerID="a325f138ceffc1f692e72c746f317f89ea3b808167515a437200034cf577a73b" Oct 30 22:30:18.314684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.314661 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:18.337151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.337124 4292 scope.go:122] "RemoveContainer" containerID="41ed39fd048fac10575b057045dd9defef8310bf9d4ee99a314577f019fe6f25" Oct 30 22:30:18.350444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.350427 4292 scope.go:122] "RemoveContainer" containerID="b18c4480cf53d730162af0d4ada3e6c83a4e7be7b377a8daca9a186be46d2aa5" Oct 30 22:30:18.362908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.362887 4292 scope.go:122] "RemoveContainer" containerID="f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572" Oct 30 22:30:18.363279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.363259 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\") pod \"08061f89-9348-4dca-ba8f-78d7447a875b\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " Oct 30 22:30:18.363514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.363492 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\" (UniqueName: \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\") pod \"08061f89-9348-4dca-ba8f-78d7447a875b\" (UID: \"08061f89-9348-4dca-ba8f-78d7447a875b\") " Oct 30 22:30:18.363642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.363595 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca" pod "08061f89-9348-4dca-ba8f-78d7447a875b" (UID: "08061f89-9348-4dca-ba8f-78d7447a875b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.364052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.364031 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08061f89-9348-4dca-ba8f-78d7447a875b-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.365565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.365536 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt" pod "08061f89-9348-4dca-ba8f-78d7447a875b" (UID: "08061f89-9348-4dca-ba8f-78d7447a875b"). InnerVolumeSpecName "kube-api-access-kmxbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.468962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.468859 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmxbt\" (UniqueName: \"kubernetes.io/projected/08061f89-9348-4dca-ba8f-78d7447a875b-kube-api-access-kmxbt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.488763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.484676 4292 scope.go:122] "RemoveContainer" containerID="f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572" Oct 30 22:30:18.489309 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:18.489271 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572\": container with ID starting with f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572 not found: ID does not exist" containerID="f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572" Oct 30 22:30:18.489431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.489320 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572"} err="failed to get container status \"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572\": rpc error: code = NotFound desc = could not find container \"f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572\": container with ID starting with f524f80a9affb5bbeafdb5326a1a34ddd0b419faaf30bac44e1a8be8eabc9572 not found: ID does not exist" Oct 30 22:30:18.489431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.489344 4292 scope.go:122] "RemoveContainer" containerID="c58249774f3536da5204b2e235c30d8201d21e452b403d354ae372f17dae18d6" Oct 30 22:30:18.496700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.496670 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:30:18.496894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.496881 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-56c628-864b8cb8dc-smz54"] Oct 30 22:30:18.530089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.530059 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:18.655074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.655037 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:30:18.657611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.657564 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-963c3a-758cd6b66c-xslvp"] Oct 30 22:30:18.902649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:18.902617 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:19.114392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.114318 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7"] Oct 30 22:30:19.115069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115070 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115087 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" Oct 30 22:30:19.115116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115093 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" Oct 30 22:30:19.115116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115106 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e217501d-0452-4108-8821-5ecb214095ee" containerName="payload-pre-processing" Oct 30 22:30:19.115116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115112 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e217501d-0452-4108-8821-5ecb214095ee" containerName="payload-pre-processing" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115123 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115128 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115136 4292 cpu_manager.go:414] "containerMap: removing container" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115141 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115161 4292 cpu_manager.go:414] "containerMap: removing container" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115166 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115177 4292 cpu_manager.go:414] "containerMap: removing container" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115181 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115190 4292 cpu_manager.go:414] "containerMap: removing container" podUID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" containerName="payload-processing" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115195 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" containerName="payload-processing" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115203 4292 cpu_manager.go:414] "containerMap: removing container" podUID="09fdea95-a4da-4d8f-a70e-504b5202bc0a" containerName="revoke-keys" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115210 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fdea95-a4da-4d8f-a70e-504b5202bc0a" containerName="revoke-keys" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115219 4292 cpu_manager.go:414] "containerMap: removing container" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" Oct 30 22:30:19.115271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115223 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115320 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115328 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e217501d-0452-4108-8821-5ecb214095ee" containerName="payload-pre-processing" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115335 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115345 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115351 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="be72a21d-df2c-4cbe-90e4-cfc491e7788c" containerName="payload-processing" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115359 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fdea95-a4da-4d8f-a70e-504b5202bc0a" containerName="revoke-keys" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115364 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" containerName="payload-pre-processing" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115374 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.115690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.115381 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" containerName="payload-processing" Oct 30 22:30:19.119044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.119028 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.138145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.138123 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:19.139659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.139631 4292 scope.go:122] "RemoveContainer" containerID="63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194" Oct 30 22:30:19.168560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.168534 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f"] Oct 30 22:30:19.169707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.169682 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.169707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.169707 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.169960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.169942 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" containerName="cleanup" Oct 30 22:30:19.170019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.169976 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" containerName="cleanup" Oct 30 22:30:19.189605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.181621 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerStarted","Data":"e04124b4845c9a634310bd257ea11d4251fbf981beb3cce6578907565eb816a6"} Oct 30 22:30:19.189605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.181746 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.189605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.182176 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191226 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l5g\" (UniqueName: \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191394 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191432 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8t29\" (UniqueName: \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191541 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191626 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.191842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.191747 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.193846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.193726 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7"] Oct 30 22:30:19.194344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.194317 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerStarted","Data":"90fd57258844811a8f73aa4a1c1acb7e2a164ba9d896244958e05c6e71225b04"} Oct 30 22:30:19.200330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.200267 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:30:19.203417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.203055 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" event={"ID":"08061f89-9348-4dca-ba8f-78d7447a875b","Type":"ContainerDied","Data":"1bf472415f2be922f8d7ac63f4df31658a4aaf000cf8576a8f87d3ebaedca8ba"} Oct 30 22:30:19.203417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.203108 4292 scope.go:122] "RemoveContainer" containerID="642da291bbf88ec014d179576fc5301647569737846de8c65d040aebda2b110a" Oct 30 22:30:19.203417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.203255 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h" Oct 30 22:30:19.208614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.206858 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-963c3a-69bf46d455-884s8"] Oct 30 22:30:19.214787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.212886 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerStarted","Data":"456925f2a582c82bfd5ca2ef2929c5d53a125ad5bbd7c3ec8bedfb1f4c70746a"} Oct 30 22:30:19.238955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.238924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerStarted","Data":"35c8cbf853954fc34e6be2af8e422d9c4aa9b2c74f1041f92f73df1022f5c2e8"} Oct 30 22:30:19.248017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.247993 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f"] Oct 30 22:30:19.266671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.261314 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:19.294626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294450 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l5g\" (UniqueName: \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.294748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294661 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.294748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294695 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8t29\" (UniqueName: \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.294882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294797 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.294882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294842 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.294993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.294920 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.299275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.299246 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.302117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.302057 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.313311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.312765 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.314733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.314313 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.361759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.361182 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l5g\" (UniqueName: \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\") pod \"maas-api-revoke-sub-0b1a8e0c-9hq5f\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.361900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.361229 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8t29\" (UniqueName: \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\") pod \"maas-api-revoke-sub-fb9cb1d4-c2hz7\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.435380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.435348 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:19.525092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.524273 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:19.653784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.653720 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:19.665365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.665332 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-56c628-29889990-hpgd9"] Oct 30 22:30:19.679051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.678956 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:19.682158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.682134 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:19.690089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.688733 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:19.691507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.691420 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:19.707605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707561 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\" (UniqueName: \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\") pod \"d2ae0e55-b704-4446-a5f8-855033b855f0\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " Oct 30 22:30:19.707691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707669 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\" (UniqueName: \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\") pod \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " Oct 30 22:30:19.707741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707725 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\") pod \"28edfe07-b99a-43e5-b724-ac304d6de0ae\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " Oct 30 22:30:19.707855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\" (UniqueName: \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\") pod \"42b99f5a-0442-454d-9e8e-a61459da0505\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " Oct 30 22:30:19.707910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707896 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\") pod \"d2ae0e55-b704-4446-a5f8-855033b855f0\" (UID: \"d2ae0e55-b704-4446-a5f8-855033b855f0\") " Oct 30 22:30:19.707945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707930 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\" (UniqueName: \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\") pod \"28edfe07-b99a-43e5-b724-ac304d6de0ae\" (UID: \"28edfe07-b99a-43e5-b724-ac304d6de0ae\") " Oct 30 22:30:19.707977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.707966 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\") pod \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\" (UID: \"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167\") " Oct 30 22:30:19.708041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.708025 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\") pod \"42b99f5a-0442-454d-9e8e-a61459da0505\" (UID: \"42b99f5a-0442-454d-9e8e-a61459da0505\") " Oct 30 22:30:19.708888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.708862 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca" pod "42b99f5a-0442-454d-9e8e-a61459da0505" (UID: "42b99f5a-0442-454d-9e8e-a61459da0505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.709759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.709734 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca" pod "d2ae0e55-b704-4446-a5f8-855033b855f0" (UID: "d2ae0e55-b704-4446-a5f8-855033b855f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.709829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.709802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca" pod "d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" (UID: "d7e8ffa0-136a-4eef-8d54-ad9e0ff89167"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.710021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.710004 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca" pod "28edfe07-b99a-43e5-b724-ac304d6de0ae" (UID: "28edfe07-b99a-43e5-b724-ac304d6de0ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.711834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.711808 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb" pod "28edfe07-b99a-43e5-b724-ac304d6de0ae" (UID: "28edfe07-b99a-43e5-b724-ac304d6de0ae"). InnerVolumeSpecName "kube-api-access-bcrjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.711922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.711876 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2" pod "d2ae0e55-b704-4446-a5f8-855033b855f0" (UID: "d2ae0e55-b704-4446-a5f8-855033b855f0"). InnerVolumeSpecName "kube-api-access-sz6m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.712639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.712598 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj" pod "42b99f5a-0442-454d-9e8e-a61459da0505" (UID: "42b99f5a-0442-454d-9e8e-a61459da0505"). InnerVolumeSpecName "kube-api-access-jgxcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.714683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.714658 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf" pod "d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" (UID: "d7e8ffa0-136a-4eef-8d54-ad9e0ff89167"). InnerVolumeSpecName "kube-api-access-bcbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.809844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809810 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edfe07-b99a-43e5-b724-ac304d6de0ae-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.809844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809847 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgxcj\" (UniqueName: \"kubernetes.io/projected/42b99f5a-0442-454d-9e8e-a61459da0505-kube-api-access-jgxcj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809862 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ae0e55-b704-4446-a5f8-855033b855f0-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809877 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrjb\" (UniqueName: \"kubernetes.io/projected/28edfe07-b99a-43e5-b724-ac304d6de0ae-kube-api-access-bcrjb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809893 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809908 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b99f5a-0442-454d-9e8e-a61459da0505-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809922 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6m2\" (UniqueName: \"kubernetes.io/projected/d2ae0e55-b704-4446-a5f8-855033b855f0-kube-api-access-sz6m2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.810026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.809936 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbsf\" (UniqueName: \"kubernetes.io/projected/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167-kube-api-access-bcbsf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.857778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.857753 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7"] Oct 30 22:30:19.858152 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:19.858131 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75f6cc3_3526_42c7_a48a_3f8aedb43017.slice/crio-359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6 WatchSource:0}: Error finding container 359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6: Status 404 returned error can't find the container with id 359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6 Oct 30 22:30:19.969443 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:19.969409 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8605ca55_a3b0_4466_8d3d_a56182d9a13a.slice/crio-e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561 WatchSource:0}: Error finding container e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561: Status 404 returned error can't find the container with id e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561 Oct 30 22:30:19.969807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:19.969588 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f"] Oct 30 22:30:20.141184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.141150 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:20.154007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.153933 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a67acb1-27b6-422c-9ba0-c6663b09fa46" path="/var/lib/kubelet/pods/4a67acb1-27b6-422c-9ba0-c6663b09fa46/volumes" Oct 30 22:30:20.154463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.154442 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992b0a74-db4f-450e-a47c-1d420001bf95" path="/var/lib/kubelet/pods/992b0a74-db4f-450e-a47c-1d420001bf95/volumes" Oct 30 22:30:20.154955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.154922 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9370076-ea62-4fcb-96f0-f8efd710e280" path="/var/lib/kubelet/pods/b9370076-ea62-4fcb-96f0-f8efd710e280/volumes" Oct 30 22:30:20.155672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.155648 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e217501d-0452-4108-8821-5ecb214095ee" path="/var/lib/kubelet/pods/e217501d-0452-4108-8821-5ecb214095ee/volumes" Oct 30 22:30:20.156479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.156462 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6d4d3c-29889990-tgn2h"] Oct 30 22:30:20.253792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.253765 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" event={"ID":"e75f6cc3-3526-42c7-a48a-3f8aedb43017","Type":"ContainerStarted","Data":"359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6"} Oct 30 22:30:20.255341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.255320 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" Oct 30 22:30:20.255456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.255333 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n" event={"ID":"42b99f5a-0442-454d-9e8e-a61459da0505","Type":"ContainerDied","Data":"7e4050067ca7bd0b4b4bd1791493609c2f5c4a3a293e39f374cfad98b0431d1f"} Oct 30 22:30:20.255456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.255398 4292 scope.go:122] "RemoveContainer" containerID="849c2501602daed8e5d120c87a39433c8d7558ad73915cd93b548d0b3a2ceeeb" Oct 30 22:30:20.257643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.257621 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerStarted","Data":"18378deee7417bfbcca15d61ff071d81bad07a797b0aa1ae6e665b1e6ffcff0c"} Oct 30 22:30:20.259257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.259238 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" Oct 30 22:30:20.259378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.259256 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6" event={"ID":"d2ae0e55-b704-4446-a5f8-855033b855f0","Type":"ContainerDied","Data":"afddd8c6e7aa5d94df8e88eb6b5d0fb3ada922ec70e40a40f2026298f9ab02c5"} Oct 30 22:30:20.261440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.261417 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerID="e04124b4845c9a634310bd257ea11d4251fbf981beb3cce6578907565eb816a6" exitCode=60 Oct 30 22:30:20.261549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.261501 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerDied","Data":"e04124b4845c9a634310bd257ea11d4251fbf981beb3cce6578907565eb816a6"} Oct 30 22:30:20.261921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.261902 4292 scope.go:122] "RemoveContainer" containerID="e04124b4845c9a634310bd257ea11d4251fbf981beb3cce6578907565eb816a6" Oct 30 22:30:20.262306 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:20.262268 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh_odh-ai-gateway-infra(9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" Oct 30 22:30:20.263997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.263977 4292 generic.go:356] "Generic (PLEG): container finished" podID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerID="90fd57258844811a8f73aa4a1c1acb7e2a164ba9d896244958e05c6e71225b04" exitCode=60 Oct 30 22:30:20.264088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.264044 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerDied","Data":"90fd57258844811a8f73aa4a1c1acb7e2a164ba9d896244958e05c6e71225b04"} Oct 30 22:30:20.264373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.264358 4292 scope.go:122] "RemoveContainer" containerID="90fd57258844811a8f73aa4a1c1acb7e2a164ba9d896244958e05c6e71225b04" Oct 30 22:30:20.264564 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:20.264538 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm_odh-ai-gateway-infra(9a06b9e5-7af8-4214-8e6a-888ea52d2290)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" Oct 30 22:30:20.266984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.266964 4292 generic.go:356] "Generic (PLEG): container finished" podID="530df160-469a-41ee-861c-c1e556fa2016" containerID="456925f2a582c82bfd5ca2ef2929c5d53a125ad5bbd7c3ec8bedfb1f4c70746a" exitCode=60 Oct 30 22:30:20.267056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.267034 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerDied","Data":"456925f2a582c82bfd5ca2ef2929c5d53a125ad5bbd7c3ec8bedfb1f4c70746a"} Oct 30 22:30:20.267361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.267344 4292 scope.go:122] "RemoveContainer" containerID="456925f2a582c82bfd5ca2ef2929c5d53a125ad5bbd7c3ec8bedfb1f4c70746a" Oct 30 22:30:20.267627 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:20.267602 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr_odh-ai-gateway-infra(530df160-469a-41ee-861c-c1e556fa2016)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" podUID="530df160-469a-41ee-861c-c1e556fa2016" Oct 30 22:30:20.268266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.268251 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" Oct 30 22:30:20.268266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.268258 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc" event={"ID":"d7e8ffa0-136a-4eef-8d54-ad9e0ff89167","Type":"ContainerDied","Data":"6357af2aeebc412d3250c307356ac1969fe29a1b7e70628a1f5bb460be99fcbc"} Oct 30 22:30:20.269140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.269119 4292 scope.go:122] "RemoveContainer" containerID="7e81fbc01197454142508f8c2734ad93c263c8b7fd0c6fc8929b7961d0b18a97" Oct 30 22:30:20.269644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.269620 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" event={"ID":"8605ca55-a3b0-4466-8d3d-a56182d9a13a","Type":"ContainerStarted","Data":"e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561"} Oct 30 22:30:20.271130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.271115 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" Oct 30 22:30:20.271223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.271143 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm" event={"ID":"28edfe07-b99a-43e5-b724-ac304d6de0ae","Type":"ContainerDied","Data":"b28770c6dcb92f2442dc1e2f6d6b8e82cd4a748ee35c96137ad50c59055f34c0"} Oct 30 22:30:20.273472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.273450 4292 generic.go:356] "Generic (PLEG): container finished" podID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerID="35c8cbf853954fc34e6be2af8e422d9c4aa9b2c74f1041f92f73df1022f5c2e8" exitCode=60 Oct 30 22:30:20.273616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.273525 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerDied","Data":"35c8cbf853954fc34e6be2af8e422d9c4aa9b2c74f1041f92f73df1022f5c2e8"} Oct 30 22:30:20.273807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.273794 4292 scope.go:122] "RemoveContainer" containerID="35c8cbf853954fc34e6be2af8e422d9c4aa9b2c74f1041f92f73df1022f5c2e8" Oct 30 22:30:20.274023 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:20.274003 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v_odh-ai-gateway-infra(0f8e9159-6c6e-4259-891c-d292138bf8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" Oct 30 22:30:20.283867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.283847 4292 scope.go:122] "RemoveContainer" containerID="aa3effd59995c14ea6a5fb09bd207525164f1b56772689dbb7a06b31210ff175" Oct 30 22:30:20.333987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.333955 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:20.381373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.381349 4292 scope.go:122] "RemoveContainer" containerID="78b3cd10b73f02e62ca99f295664273d2d76b05efa562679e81a836f2a17e9a0" Oct 30 22:30:20.402981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.397315 4292 scope.go:122] "RemoveContainer" containerID="0e2431518d686b957146cb357df394b0758a606bc18098ffe39129b72058f778" Oct 30 22:30:20.416411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.416384 4292 scope.go:122] "RemoveContainer" containerID="da68b32737948d64046bc314b06be8a235781d8ed0d40d653ffa7f69734e7323" Oct 30 22:30:20.428925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.428876 4292 scope.go:122] "RemoveContainer" containerID="1b441e880aa4c169d2748372f7dadb107f99d25f23e9a6f5df0305048f4c8dda" Oct 30 22:30:20.439949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.439929 4292 scope.go:122] "RemoveContainer" containerID="4168f7e9e62da910bfa3b846ba7669dbb0bfcbfb26cd45dac6c946fcc3967b71" Oct 30 22:30:20.501271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.501247 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:20.525147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.525123 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\") pod \"e12303c3-00b0-4d84-b007-a3f040b44ea6\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " Oct 30 22:30:20.525253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.525169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\" (UniqueName: \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\") pod \"e12303c3-00b0-4d84-b007-a3f040b44ea6\" (UID: \"e12303c3-00b0-4d84-b007-a3f040b44ea6\") " Oct 30 22:30:20.525464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.525444 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca" pod "e12303c3-00b0-4d84-b007-a3f040b44ea6" (UID: "e12303c3-00b0-4d84-b007-a3f040b44ea6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.525869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.525852 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12303c3-00b0-4d84-b007-a3f040b44ea6-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.527379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.527358 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk" pod "e12303c3-00b0-4d84-b007-a3f040b44ea6" (UID: "e12303c3-00b0-4d84-b007-a3f040b44ea6"). InnerVolumeSpecName "kube-api-access-m4tqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.627036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.626995 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4tqk\" (UniqueName: \"kubernetes.io/projected/e12303c3-00b0-4d84-b007-a3f040b44ea6-kube-api-access-m4tqk\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.845983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:20.845932 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" podStartSLOduration=20.845917566 podStartE2EDuration="20.845917566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:20.841758569 +0000 UTC m=+2148.810578134" watchObservedRunningTime="2026-10-30 22:30:20.845917566 +0000 UTC m=+2148.814737130" Oct 30 22:30:21.023942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.023912 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:21.275254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.275169 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:21.283403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.283376 4292 generic.go:356] "Generic (PLEG): container finished" podID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerID="18378deee7417bfbcca15d61ff071d81bad07a797b0aa1ae6e665b1e6ffcff0c" exitCode=60 Oct 30 22:30:21.283538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.283447 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerDied","Data":"18378deee7417bfbcca15d61ff071d81bad07a797b0aa1ae6e665b1e6ffcff0c"} Oct 30 22:30:21.283538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.283507 4292 scope.go:122] "RemoveContainer" containerID="63fbaceb702a8625f9526d3ce683dd2e7f9c931e46ecab22bd2479659ed98194" Oct 30 22:30:21.283953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.283933 4292 scope.go:122] "RemoveContainer" containerID="18378deee7417bfbcca15d61ff071d81bad07a797b0aa1ae6e665b1e6ffcff0c" Oct 30 22:30:21.284244 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:21.284215 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks_odh-ai-gateway-infra(34d2a0e3-b6dc-417f-89da-4fd53c19b981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" Oct 30 22:30:21.289762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.289730 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" Oct 30 22:30:21.289894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.289768 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp" event={"ID":"e12303c3-00b0-4d84-b007-a3f040b44ea6","Type":"ContainerDied","Data":"1f092b1660692d13075785a86874451d5ca638bb8f6005112e8ea7e90dc72400"} Oct 30 22:30:21.294858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.294834 4292 generic.go:356] "Generic (PLEG): container finished" podID="8605ca55-a3b0-4466-8d3d-a56182d9a13a" containerID="fe8ff60fbc8dcf156981639536dbc95804110e512826f525a74ed137a317675f" exitCode=0 Oct 30 22:30:21.294985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.294967 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" event={"ID":"8605ca55-a3b0-4466-8d3d-a56182d9a13a","Type":"ContainerDied","Data":"fe8ff60fbc8dcf156981639536dbc95804110e512826f525a74ed137a317675f"} Oct 30 22:30:21.299061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.299036 4292 generic.go:356] "Generic (PLEG): container finished" podID="e75f6cc3-3526-42c7-a48a-3f8aedb43017" containerID="35d513db3787dbb5d3f2b04e2420db17084f930a091c98a52ff2a2d767417f70" exitCode=0 Oct 30 22:30:21.299160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.299111 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" event={"ID":"e75f6cc3-3526-42c7-a48a-3f8aedb43017","Type":"ContainerDied","Data":"35d513db3787dbb5d3f2b04e2420db17084f930a091c98a52ff2a2d767417f70"} Oct 30 22:30:21.315596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.315468 4292 scope.go:122] "RemoveContainer" containerID="b8c57db269f35cdd21e8fad10cc5ab48405d2ff35bcf854ee62da79293e2ad2a" Oct 30 22:30:21.328230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.328186 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:21.334489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.334463 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d50b13-29889990-7n8w6"] Oct 30 22:30:21.477986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.477962 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:21.506044 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.506017 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:21.539405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.539341 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\") pod \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " Oct 30 22:30:21.539497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.539447 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\" (UniqueName: \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\") pod \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\" (UID: \"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc\") " Oct 30 22:30:21.539679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.539563 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\") pod \"530df160-469a-41ee-861c-c1e556fa2016\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " Oct 30 22:30:21.539872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.539800 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\" (UniqueName: \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\") pod \"530df160-469a-41ee-861c-c1e556fa2016\" (UID: \"530df160-469a-41ee-861c-c1e556fa2016\") " Oct 30 22:30:21.540001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.539919 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca" pod "530df160-469a-41ee-861c-c1e556fa2016" (UID: "530df160-469a-41ee-861c-c1e556fa2016"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.540281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.540260 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/530df160-469a-41ee-861c-c1e556fa2016-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.541184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.541159 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca" pod "9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" (UID: "9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.542073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.542053 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj" pod "9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" (UID: "9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc"). InnerVolumeSpecName "kube-api-access-62lpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.542219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.542199 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh" pod "530df160-469a-41ee-861c-c1e556fa2016" (UID: "530df160-469a-41ee-861c-c1e556fa2016"). InnerVolumeSpecName "kube-api-access-hh9wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.641561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.641531 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9wh\" (UniqueName: \"kubernetes.io/projected/530df160-469a-41ee-861c-c1e556fa2016-kube-api-access-hh9wh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.641561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.641558 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.641737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.641569 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62lpj\" (UniqueName: \"kubernetes.io/projected/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc-kube-api-access-62lpj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.799985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.799952 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:21.812703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.812676 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18515933-29889990-gnzzm"] Oct 30 22:30:21.858799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.858768 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:21.864086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.864040 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" podUID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" containerName="payload-processing" containerID="cri-o://3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b" gracePeriod=25 Oct 30 22:30:21.931429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.931367 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:30:21.937601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.937521 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:21.943286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.943263 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-b7588a-29889990-q5pmc"] Oct 30 22:30:21.977281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.977247 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" podUID="901c94fa-a358-4834-80a9-7a980f08703c" containerName="payload-pre-processing" containerID="cri-o://5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43" gracePeriod=25 Oct 30 22:30:21.993304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:21.993284 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:22.049482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.049446 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\") pod \"0f8e9159-6c6e-4259-891c-d292138bf8d3\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " Oct 30 22:30:22.049938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.049550 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\" (UniqueName: \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\") pod \"0f8e9159-6c6e-4259-891c-d292138bf8d3\" (UID: \"0f8e9159-6c6e-4259-891c-d292138bf8d3\") " Oct 30 22:30:22.051181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.051110 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca" pod "0f8e9159-6c6e-4259-891c-d292138bf8d3" (UID: "0f8e9159-6c6e-4259-891c-d292138bf8d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.052872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.052769 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:30:22.058746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.058482 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j" pod "0f8e9159-6c6e-4259-891c-d292138bf8d3" (UID: "0f8e9159-6c6e-4259-891c-d292138bf8d3"). InnerVolumeSpecName "kube-api-access-tcn5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.060658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.060636 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:22.101334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.101306 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-x4k9n"] Oct 30 22:30:22.106617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.106545 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:22.146407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.146374 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:30:22.151537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.151501 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e9159-6c6e-4259-891c-d292138bf8d3-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.151687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.151539 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcn5j\" (UniqueName: \"kubernetes.io/projected/0f8e9159-6c6e-4259-891c-d292138bf8d3-kube-api-access-tcn5j\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.154887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.154859 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08061f89-9348-4dca-ba8f-78d7447a875b" path="/var/lib/kubelet/pods/08061f89-9348-4dca-ba8f-78d7447a875b/volumes" Oct 30 22:30:22.155328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.155311 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" path="/var/lib/kubelet/pods/28edfe07-b99a-43e5-b724-ac304d6de0ae/volumes" Oct 30 22:30:22.155723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.155705 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" path="/var/lib/kubelet/pods/42b99f5a-0442-454d-9e8e-a61459da0505/volumes" Oct 30 22:30:22.156501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.156481 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" path="/var/lib/kubelet/pods/d2ae0e55-b704-4446-a5f8-855033b855f0/volumes" Oct 30 22:30:22.156890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.156875 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" path="/var/lib/kubelet/pods/d7e8ffa0-136a-4eef-8d54-ad9e0ff89167/volumes" Oct 30 22:30:22.252376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.252343 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\") pod \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " Oct 30 22:30:22.252552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.252419 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\" (UniqueName: \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\") pod \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\" (UID: \"9903bac3-efc9-4de3-b4ea-e84a7fc853da\") " Oct 30 22:30:22.260064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.260032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh" pod "9903bac3-efc9-4de3-b4ea-e84a7fc853da" (UID: "9903bac3-efc9-4de3-b4ea-e84a7fc853da"). InnerVolumeSpecName "kube-api-access-zfrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.266094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.266068 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume" pod "9903bac3-efc9-4de3-b4ea-e84a7fc853da" (UID: "9903bac3-efc9-4de3-b4ea-e84a7fc853da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.274008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.273984 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:30:22.315417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.315388 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" event={"ID":"9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc","Type":"ContainerDied","Data":"541ea9759952decfca7da31f4a85e41fe3489c1312f850b0626850de33c43a65"} Oct 30 22:30:22.315417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.315409 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh" Oct 30 22:30:22.315607 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.315456 4292 scope.go:122] "RemoveContainer" containerID="e04124b4845c9a634310bd257ea11d4251fbf981beb3cce6578907565eb816a6" Oct 30 22:30:22.317069 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.317048 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" event={"ID":"9a06b9e5-7af8-4214-8e6a-888ea52d2290","Type":"ContainerDied","Data":"f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8"} Oct 30 22:30:22.317161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.317094 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b8a4179af7aed5e85fe04765472e0e4e9f8809bc2990cd6cc61309a960abc8" Oct 30 22:30:22.320007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.319720 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" Oct 30 22:30:22.320007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.319732 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr" event={"ID":"530df160-469a-41ee-861c-c1e556fa2016","Type":"ContainerDied","Data":"99d275a41b81b690d63d5d98443040070230e35593cb8dc21711d477cbb69b1a"} Oct 30 22:30:22.321144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.321125 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" Oct 30 22:30:22.321224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.321127 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v" event={"ID":"0f8e9159-6c6e-4259-891c-d292138bf8d3","Type":"ContainerDied","Data":"92d5b53d60ead7108a08763f6e199e26a1417b37665ef190f709cc65d6b82608"} Oct 30 22:30:22.322491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.322472 4292 generic.go:356] "Generic (PLEG): container finished" podID="901c94fa-a358-4834-80a9-7a980f08703c" containerID="5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43" exitCode=0 Oct 30 22:30:22.322622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.322516 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" Oct 30 22:30:22.322622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.322554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" event={"ID":"901c94fa-a358-4834-80a9-7a980f08703c","Type":"ContainerDied","Data":"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43"} Oct 30 22:30:22.322622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.322612 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g" event={"ID":"901c94fa-a358-4834-80a9-7a980f08703c","Type":"ContainerDied","Data":"c595294687ca3bd3105e019e98377da8a9b93558297f681a30f9cd12d12a09c4"} Oct 30 22:30:22.323823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.323804 4292 generic.go:356] "Generic (PLEG): container finished" podID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" containerID="3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b" exitCode=0 Oct 30 22:30:22.323913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.323842 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" Oct 30 22:30:22.323913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.323874 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" event={"ID":"9903bac3-efc9-4de3-b4ea-e84a7fc853da","Type":"ContainerDied","Data":"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b"} Oct 30 22:30:22.323913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.323910 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz" event={"ID":"9903bac3-efc9-4de3-b4ea-e84a7fc853da","Type":"ContainerDied","Data":"525291b5555bc534694274469afc9705ce274aa282df31176288a7cdd41741ad"} Oct 30 22:30:22.355439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.354421 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\" (UniqueName: \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\") pod \"901c94fa-a358-4834-80a9-7a980f08703c\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " Oct 30 22:30:22.355439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.354569 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\") pod \"901c94fa-a358-4834-80a9-7a980f08703c\" (UID: \"901c94fa-a358-4834-80a9-7a980f08703c\") " Oct 30 22:30:22.355439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.355197 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903bac3-efc9-4de3-b4ea-e84a7fc853da-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.355439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.355220 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrnh\" (UniqueName: \"kubernetes.io/projected/9903bac3-efc9-4de3-b4ea-e84a7fc853da-kube-api-access-zfrnh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.357368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.355984 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume" pod "901c94fa-a358-4834-80a9-7a980f08703c" (UID: "901c94fa-a358-4834-80a9-7a980f08703c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.358192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.357695 4292 scope.go:122] "RemoveContainer" containerID="456925f2a582c82bfd5ca2ef2929c5d53a125ad5bbd7c3ec8bedfb1f4c70746a" Oct 30 22:30:22.361477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.361426 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:22.367879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.367839 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns" pod "901c94fa-a358-4834-80a9-7a980f08703c" (UID: "901c94fa-a358-4834-80a9-7a980f08703c"). InnerVolumeSpecName "kube-api-access-9x9ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.415782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.414990 4292 scope.go:122] "RemoveContainer" containerID="35c8cbf853954fc34e6be2af8e422d9c4aa9b2c74f1041f92f73df1022f5c2e8" Oct 30 22:30:22.436939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.435641 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:22.439078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.439054 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-77cb1f-29889990-bqftp"] Oct 30 22:30:22.443563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.443543 4292 scope.go:122] "RemoveContainer" containerID="5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43" Oct 30 22:30:22.457133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.456177 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\") pod \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " Oct 30 22:30:22.457133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.456255 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\" (UniqueName: \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\") pod \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\" (UID: \"9a06b9e5-7af8-4214-8e6a-888ea52d2290\") " Oct 30 22:30:22.457133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.456727 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x9ns\" (UniqueName: \"kubernetes.io/projected/901c94fa-a358-4834-80a9-7a980f08703c-kube-api-access-9x9ns\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.457133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.456740 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c94fa-a358-4834-80a9-7a980f08703c-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.457390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.457204 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca" pod "9a06b9e5-7af8-4214-8e6a-888ea52d2290" (UID: "9a06b9e5-7af8-4214-8e6a-888ea52d2290"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.458259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.458240 4292 scope.go:122] "RemoveContainer" containerID="5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43" Oct 30 22:30:22.458544 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:22.458519 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43\": container with ID starting with 5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43 not found: ID does not exist" containerID="5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43" Oct 30 22:30:22.458667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.458549 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43"} err="failed to get container status \"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43\": rpc error: code = NotFound desc = could not find container \"5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43\": container with ID starting with 5bd1d8e3077d3f2ff738f90898df5678b064a6d847f2d353a5c55a67686a2c43 not found: ID does not exist" Oct 30 22:30:22.458667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.458566 4292 scope.go:122] "RemoveContainer" containerID="3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b" Oct 30 22:30:22.460304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.460191 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh" pod "9a06b9e5-7af8-4214-8e6a-888ea52d2290" (UID: "9a06b9e5-7af8-4214-8e6a-888ea52d2290"). InnerVolumeSpecName "kube-api-access-f8jvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.499041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.498972 4292 scope.go:122] "RemoveContainer" containerID="3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b" Oct 30 22:30:22.499523 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:22.499500 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b\": container with ID starting with 3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b not found: ID does not exist" containerID="3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b" Oct 30 22:30:22.499642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.499532 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b"} err="failed to get container status \"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b\": rpc error: code = NotFound desc = could not find container \"3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b\": container with ID starting with 3f43495a6d06eee2080d3a63ddfee6c8ea6fad439da0edf3b89ccaed8da7ff4b not found: ID does not exist" Oct 30 22:30:22.566832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.566669 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a06b9e5-7af8-4214-8e6a-888ea52d2290-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.566832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.566703 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jvh\" (UniqueName: \"kubernetes.io/projected/9a06b9e5-7af8-4214-8e6a-888ea52d2290-kube-api-access-f8jvh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.585427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.585401 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:22.589647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.589627 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:22.617509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.617482 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669167 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\" (UniqueName: \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\") pod \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669321 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\") pod \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669378 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\") pod \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669550 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\" (UniqueName: \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\") pod \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\") pod \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\" (UID: \"8605ca55-a3b0-4466-8d3d-a56182d9a13a\") " Oct 30 22:30:22.669916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669895 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca" pod "e75f6cc3-3526-42c7-a48a-3f8aedb43017" (UID: "e75f6cc3-3526-42c7-a48a-3f8aedb43017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.670342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.669940 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca" pod "8605ca55-a3b0-4466-8d3d-a56182d9a13a" (UID: "8605ca55-a3b0-4466-8d3d-a56182d9a13a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.670342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.670014 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\") pod \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\" (UID: \"e75f6cc3-3526-42c7-a48a-3f8aedb43017\") " Oct 30 22:30:22.671914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.670673 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.671914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.670704 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.673170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.673130 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29" pod "e75f6cc3-3526-42c7-a48a-3f8aedb43017" (UID: "e75f6cc3-3526-42c7-a48a-3f8aedb43017"). InnerVolumeSpecName "kube-api-access-x8t29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.674339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.674296 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca" pod "8605ca55-a3b0-4466-8d3d-a56182d9a13a" (UID: "8605ca55-a3b0-4466-8d3d-a56182d9a13a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:22.675209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.675174 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g" pod "8605ca55-a3b0-4466-8d3d-a56182d9a13a" (UID: "8605ca55-a3b0-4466-8d3d-a56182d9a13a"). InnerVolumeSpecName "kube-api-access-29l5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.675381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.675367 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca" pod "e75f6cc3-3526-42c7-a48a-3f8aedb43017" (UID: "e75f6cc3-3526-42c7-a48a-3f8aedb43017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:22.758344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.758320 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:22.772010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.771988 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8t29\" (UniqueName: \"kubernetes.io/projected/e75f6cc3-3526-42c7-a48a-3f8aedb43017-kube-api-access-x8t29\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.772010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.772011 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8605ca55-a3b0-4466-8d3d-a56182d9a13a-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.772141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.772022 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f6cc3-3526-42c7-a48a-3f8aedb43017-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.772141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.772032 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29l5g\" (UniqueName: \"kubernetes.io/projected/8605ca55-a3b0-4466-8d3d-a56182d9a13a-kube-api-access-29l5g\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.872690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.872625 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\" (UniqueName: \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\") pod \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " Oct 30 22:30:22.872799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.872766 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\") pod \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\" (UID: \"34d2a0e3-b6dc-417f-89da-4fd53c19b981\") " Oct 30 22:30:22.873094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.873072 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca" pod "34d2a0e3-b6dc-417f-89da-4fd53c19b981" (UID: "34d2a0e3-b6dc-417f-89da-4fd53c19b981"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.874603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.874569 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj" pod "34d2a0e3-b6dc-417f-89da-4fd53c19b981" (UID: "34d2a0e3-b6dc-417f-89da-4fd53c19b981"). InnerVolumeSpecName "kube-api-access-6c5lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.974013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.973985 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d2a0e3-b6dc-417f-89da-4fd53c19b981-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.974013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:22.974012 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5lj\" (UniqueName: \"kubernetes.io/projected/34d2a0e3-b6dc-417f-89da-4fd53c19b981-kube-api-access-6c5lj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.345469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.345437 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" Oct 30 22:30:23.345469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.345447 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7" event={"ID":"e75f6cc3-3526-42c7-a48a-3f8aedb43017","Type":"ContainerDied","Data":"359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6"} Oct 30 22:30:23.346064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.345520 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="359efab67cafe54f4aef41e1783ee88d58867f4bfb34f9a031215df6adc8d0d6" Oct 30 22:30:23.356436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.356407 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" Oct 30 22:30:23.356705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.356680 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks" event={"ID":"34d2a0e3-b6dc-417f-89da-4fd53c19b981","Type":"ContainerDied","Data":"feb42c16f764763767154dfcd1e40c27c65426d187c4ed83dee0feae9b96eb88"} Oct 30 22:30:23.356822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.356745 4292 scope.go:122] "RemoveContainer" containerID="18378deee7417bfbcca15d61ff071d81bad07a797b0aa1ae6e665b1e6ffcff0c" Oct 30 22:30:23.364832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.364812 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" Oct 30 22:30:23.364929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.364877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f" event={"ID":"8605ca55-a3b0-4466-8d3d-a56182d9a13a","Type":"ContainerDied","Data":"e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561"} Oct 30 22:30:23.365010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.364928 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e8e6931ba6122432f6cc50858a0fad0271b25923c65d90dfb78e23dd00d561" Oct 30 22:30:23.372168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:23.372143 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm" Oct 30 22:30:24.002251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.002220 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:30:24.010721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.010696 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6d4d3c-7b7765f547-mwszz"] Oct 30 22:30:24.122891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.122864 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:24.132720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.132694 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c76d9e-29889990-xn5nh"] Oct 30 22:30:24.148150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.148116 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" path="/var/lib/kubelet/pods/9903bac3-efc9-4de3-b4ea-e84a7fc853da/volumes" Oct 30 22:30:24.148640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.148619 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" path="/var/lib/kubelet/pods/9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc/volumes" Oct 30 22:30:24.149078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.149057 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" path="/var/lib/kubelet/pods/e12303c3-00b0-4d84-b007-a3f040b44ea6/volumes" Oct 30 22:30:24.310382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.310355 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:24.320872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.320847 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2a4123-29889990-9ksks"] Oct 30 22:30:24.433483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.433455 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:30:24.437509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.437489 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6d4d3c-5df5b77b59-dhk4g"] Oct 30 22:30:24.550644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.550610 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:24.556279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.556252 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-963c3a-29889990-l99fr"] Oct 30 22:30:24.842598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.823620 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:24.842598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:24.833966 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-97aa64-29889990-fcl7v"] Oct 30 22:30:25.103714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:25.103638 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:25.108797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:25.108771 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6c7d3f-29889990-v88lm"] Oct 30 22:30:26.147994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.147956 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" path="/var/lib/kubelet/pods/0f8e9159-6c6e-4259-891c-d292138bf8d3/volumes" Oct 30 22:30:26.148401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.148386 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" path="/var/lib/kubelet/pods/34d2a0e3-b6dc-417f-89da-4fd53c19b981/volumes" Oct 30 22:30:26.148802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.148787 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530df160-469a-41ee-861c-c1e556fa2016" path="/var/lib/kubelet/pods/530df160-469a-41ee-861c-c1e556fa2016/volumes" Oct 30 22:30:26.149448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.149433 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901c94fa-a358-4834-80a9-7a980f08703c" path="/var/lib/kubelet/pods/901c94fa-a358-4834-80a9-7a980f08703c/volumes" Oct 30 22:30:26.149811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.149797 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" path="/var/lib/kubelet/pods/9a06b9e5-7af8-4214-8e6a-888ea52d2290/volumes" Oct 30 22:30:26.682225 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:26.682184 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:30:26.932547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:26.932474 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" podUID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" containerName="payload-processing" containerID="cri-o://5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817" gracePeriod=25 Oct 30 22:30:27.055348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.055314 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" podUID="0cbbcbc0-31b4-4050-b556-c12e61da789b" containerName="payload-pre-processing" containerID="cri-o://50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428" gracePeriod=25 Oct 30 22:30:27.214791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.214757 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:30:27.255988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.255959 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\" (UniqueName: \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\") pod \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " Oct 30 22:30:27.256147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.256091 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\") pod \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\" (UID: \"9a1b3c09-95a5-4a94-a674-77d7ddee3482\") " Oct 30 22:30:27.256455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.256430 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume" pod "9a1b3c09-95a5-4a94-a674-77d7ddee3482" (UID: "9a1b3c09-95a5-4a94-a674-77d7ddee3482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:27.256667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.256651 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1b3c09-95a5-4a94-a674-77d7ddee3482-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:27.258125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.258107 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh" pod "9a1b3c09-95a5-4a94-a674-77d7ddee3482" (UID: "9a1b3c09-95a5-4a94-a674-77d7ddee3482"). InnerVolumeSpecName "kube-api-access-gpwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:27.327792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.327767 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:30:27.357202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.357175 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\") pod \"0cbbcbc0-31b4-4050-b556-c12e61da789b\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " Oct 30 22:30:27.357312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.357295 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\" (UniqueName: \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\") pod \"0cbbcbc0-31b4-4050-b556-c12e61da789b\" (UID: \"0cbbcbc0-31b4-4050-b556-c12e61da789b\") " Oct 30 22:30:27.357695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.357667 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume" pod "0cbbcbc0-31b4-4050-b556-c12e61da789b" (UID: "0cbbcbc0-31b4-4050-b556-c12e61da789b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:27.357903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.357882 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwjh\" (UniqueName: \"kubernetes.io/projected/9a1b3c09-95a5-4a94-a674-77d7ddee3482-kube-api-access-gpwjh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:27.358005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.357911 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbbcbc0-31b4-4050-b556-c12e61da789b-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:27.359700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.359672 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq" pod "0cbbcbc0-31b4-4050-b556-c12e61da789b" (UID: "0cbbcbc0-31b4-4050-b556-c12e61da789b"). InnerVolumeSpecName "kube-api-access-qhnsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:27.406250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.406217 4292 generic.go:356] "Generic (PLEG): container finished" podID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" containerID="5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817" exitCode=0 Oct 30 22:30:27.406393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.406281 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" Oct 30 22:30:27.406393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.406318 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" event={"ID":"9a1b3c09-95a5-4a94-a674-77d7ddee3482","Type":"ContainerDied","Data":"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817"} Oct 30 22:30:27.406393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.406373 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5" event={"ID":"9a1b3c09-95a5-4a94-a674-77d7ddee3482","Type":"ContainerDied","Data":"a0aaedc2a27270cb47ab020c9e92f4f02cbb8baf60bece3c71fbd39242108f65"} Oct 30 22:30:27.406585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.406398 4292 scope.go:122] "RemoveContainer" containerID="5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817" Oct 30 22:30:27.408742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.408713 4292 generic.go:356] "Generic (PLEG): container finished" podID="0cbbcbc0-31b4-4050-b556-c12e61da789b" containerID="50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428" exitCode=0 Oct 30 22:30:27.408857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.408768 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" Oct 30 22:30:27.408857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.408819 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" event={"ID":"0cbbcbc0-31b4-4050-b556-c12e61da789b","Type":"ContainerDied","Data":"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428"} Oct 30 22:30:27.408979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.408864 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6" event={"ID":"0cbbcbc0-31b4-4050-b556-c12e61da789b","Type":"ContainerDied","Data":"d229db41e7dfff8201bcfc1db15ed12f185d00611e6c91842f36fd5585d06d4b"} Oct 30 22:30:27.423179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.423158 4292 scope.go:122] "RemoveContainer" containerID="5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817" Oct 30 22:30:27.423463 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:27.423443 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817\": container with ID starting with 5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817 not found: ID does not exist" containerID="5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817" Oct 30 22:30:27.423525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.423469 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817"} err="failed to get container status \"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817\": rpc error: code = NotFound desc = could not find container \"5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817\": container with ID starting with 5a09731c218ed508cab726b0179056e7813e6d28f4fb322d645954c4919d6817 not found: ID does not exist" Oct 30 22:30:27.423525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.423485 4292 scope.go:122] "RemoveContainer" containerID="50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428" Oct 30 22:30:27.433237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.433221 4292 scope.go:122] "RemoveContainer" containerID="50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428" Oct 30 22:30:27.433493 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:27.433471 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428\": container with ID starting with 50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428 not found: ID does not exist" containerID="50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428" Oct 30 22:30:27.433536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.433500 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428"} err="failed to get container status \"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428\": rpc error: code = NotFound desc = could not find container \"50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428\": container with ID starting with 50e6e0de9b9d29adef66ce300c79423c2a730ebc7f7d7f9c807e86bd4e8ce428 not found: ID does not exist" Oct 30 22:30:27.458717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.458695 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhnsq\" (UniqueName: \"kubernetes.io/projected/0cbbcbc0-31b4-4050-b556-c12e61da789b-kube-api-access-qhnsq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:27.597745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.597705 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:30:27.602387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.602361 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-53cceb-5866c77c45-9z9t5"] Oct 30 22:30:27.715016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.714957 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:30:27.722076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:27.722051 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-53cceb-647bfb8df5-gx7n6"] Oct 30 22:30:28.147449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:28.147419 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbbcbc0-31b4-4050-b556-c12e61da789b" path="/var/lib/kubelet/pods/0cbbcbc0-31b4-4050-b556-c12e61da789b/volumes" Oct 30 22:30:28.147873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:28.147858 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" path="/var/lib/kubelet/pods/9a1b3c09-95a5-4a94-a674-77d7ddee3482/volumes" Oct 30 22:30:32.770122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:32.770090 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p"] Oct 30 22:30:32.775741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:32.775708 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afe802-m9s5p"] Oct 30 22:30:34.147222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:34.147189 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f862b947-3965-4ed3-bcfe-756bfcbb2487" path="/var/lib/kubelet/pods/f862b947-3965-4ed3-bcfe-756bfcbb2487/volumes" Oct 30 22:30:40.868404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:40.868368 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:30:40.868840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:40.868589 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" containerID="cri-o://4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545" gracePeriod=30 Oct 30 22:30:42.292783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.292746 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8"] Oct 30 22:30:42.293313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293298 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" containerName="payload-processing" Oct 30 22:30:42.293355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293314 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" containerName="payload-processing" Oct 30 22:30:42.293355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293331 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.293355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293336 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.293355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293345 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.293355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293349 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293358 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" containerName="payload-processing" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293362 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" containerName="payload-processing" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293371 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293375 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293381 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293386 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293402 4292 cpu_manager.go:414] "containerMap: removing container" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293409 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293425 4292 cpu_manager.go:414] "containerMap: removing container" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293433 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293443 4292 cpu_manager.go:414] "containerMap: removing container" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293450 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293457 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e75f6cc3-3526-42c7-a48a-3f8aedb43017" containerName="revoke-keys" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293461 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75f6cc3-3526-42c7-a48a-3f8aedb43017" containerName="revoke-keys" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293468 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293473 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293482 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293486 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293492 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0cbbcbc0-31b4-4050-b556-c12e61da789b" containerName="payload-pre-processing" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293497 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbbcbc0-31b4-4050-b556-c12e61da789b" containerName="payload-pre-processing" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293512 4292 cpu_manager.go:414] "containerMap: removing container" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293516 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.293515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293522 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293526 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293533 4292 cpu_manager.go:414] "containerMap: removing container" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293537 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293542 4292 cpu_manager.go:414] "containerMap: removing container" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293546 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293552 4292 cpu_manager.go:414] "containerMap: removing container" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293556 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293563 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293567 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293584 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293590 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293601 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293605 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293618 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293623 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293628 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293632 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293639 4292 cpu_manager.go:414] "containerMap: removing container" podUID="901c94fa-a358-4834-80a9-7a980f08703c" containerName="payload-pre-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293643 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901c94fa-a358-4834-80a9-7a980f08703c" containerName="payload-pre-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293649 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293653 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293660 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293664 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293672 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8605ca55-a3b0-4466-8d3d-a56182d9a13a" containerName="revoke-keys" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293676 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8605ca55-a3b0-4466-8d3d-a56182d9a13a" containerName="revoke-keys" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293683 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293687 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293696 4292 cpu_manager.go:414] "containerMap: removing container" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293700 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293706 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293710 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293716 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293720 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293729 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293733 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293831 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293840 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293846 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="901c94fa-a358-4834-80a9-7a980f08703c" containerName="payload-pre-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293853 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293859 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293865 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8605ca55-a3b0-4466-8d3d-a56182d9a13a" containerName="revoke-keys" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293871 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293877 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293882 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293886 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293892 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293896 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a06b9e5-7af8-4214-8e6a-888ea52d2290" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293902 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293907 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9903bac3-efc9-4de3-b4ea-e84a7fc853da" containerName="payload-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293913 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293919 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293923 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1b3c09-95a5-4a94-a674-77d7ddee3482" containerName="payload-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293930 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293936 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293941 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293946 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75f6cc3-3526-42c7-a48a-3f8aedb43017" containerName="revoke-keys" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293951 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293956 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293962 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbbcbc0-31b4-4050-b556-c12e61da789b" containerName="payload-pre-processing" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293969 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293974 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293980 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293986 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:30:42.294354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.293991 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e8ffa0-136a-4eef-8d54-ad9e0ff89167" containerName="cleanup" Oct 30 22:30:42.299646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.299628 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.355960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.355933 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8"] Oct 30 22:30:42.402270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.402246 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:42.402435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.402415 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:30:42.402492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.402419 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:42.403891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.403874 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:42.438954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.438920 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.439076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.439056 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.439156 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.439136 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdww\" (UniqueName: \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.540368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.540340 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.540475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.540417 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.540475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.540452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdww\" (UniqueName: \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.541102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.541075 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.542917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.542870 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.604118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.604090 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdww\" (UniqueName: \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\") pod \"maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.612906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.612882 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:42.992989 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:30:42.992961 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b619ea_60fd_41de_9b26_bbd2c98de9a3.slice/crio-ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312 WatchSource:0}: Error finding container ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312: Status 404 returned error can't find the container with id ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312 Oct 30 22:30:42.993102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:42.993041 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8"] Oct 30 22:30:43.555719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:43.555684 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" event={"ID":"64b619ea-60fd-41de-9b26-bbd2c98de9a3","Type":"ContainerStarted","Data":"67b159a8ff160664a130cbe6b8ac6aac27e0af021cdb5489db71a3be56f46c90"} Oct 30 22:30:43.556118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:43.555741 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" event={"ID":"64b619ea-60fd-41de-9b26-bbd2c98de9a3","Type":"ContainerStarted","Data":"ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312"} Oct 30 22:30:43.676587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:43.676525 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" podStartSLOduration=1.676506159 podStartE2EDuration="1.676506159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:43.674421647 +0000 UTC m=+2171.643241215" watchObservedRunningTime="2026-10-30 22:30:43.676506159 +0000 UTC m=+2171.645325720" Oct 30 22:30:44.563700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:44.563669 4292 generic.go:356] "Generic (PLEG): container finished" podID="64b619ea-60fd-41de-9b26-bbd2c98de9a3" containerID="67b159a8ff160664a130cbe6b8ac6aac27e0af021cdb5489db71a3be56f46c90" exitCode=0 Oct 30 22:30:44.564166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:44.563700 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" event={"ID":"64b619ea-60fd-41de-9b26-bbd2c98de9a3","Type":"ContainerDied","Data":"67b159a8ff160664a130cbe6b8ac6aac27e0af021cdb5489db71a3be56f46c90"} Oct 30 22:30:45.717648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.717620 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:45.880197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.880090 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\") pod \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " Oct 30 22:30:45.880291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.880273 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\" (UniqueName: \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\") pod \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " Oct 30 22:30:45.880388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.880368 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\") pod \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\" (UID: \"64b619ea-60fd-41de-9b26-bbd2c98de9a3\") " Oct 30 22:30:45.881001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.880975 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca" pod "64b619ea-60fd-41de-9b26-bbd2c98de9a3" (UID: "64b619ea-60fd-41de-9b26-bbd2c98de9a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:45.882898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.882871 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww" pod "64b619ea-60fd-41de-9b26-bbd2c98de9a3" (UID: "64b619ea-60fd-41de-9b26-bbd2c98de9a3"). InnerVolumeSpecName "kube-api-access-9mdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:45.883035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.882932 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca" pod "64b619ea-60fd-41de-9b26-bbd2c98de9a3" (UID: "64b619ea-60fd-41de-9b26-bbd2c98de9a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:45.982005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.981970 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mdww\" (UniqueName: \"kubernetes.io/projected/64b619ea-60fd-41de-9b26-bbd2c98de9a3-kube-api-access-9mdww\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:45.982005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.982001 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:45.982168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:45.982043 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b619ea-60fd-41de-9b26-bbd2c98de9a3-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.142398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.142371 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:30:46.285387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285354 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285414 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285467 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285502 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285692 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285779 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285820 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.285986 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285958 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.286041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.286014 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\") pod \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\" (UID: \"0f23de09-a8c2-44ce-99a2-f49ef3cb015a\") " Oct 30 22:30:46.286118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285819 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:46.286209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.285970 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:46.287163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.286779 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.287163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.286811 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.287568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.287528 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:46.287754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.287721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:46.287947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.287919 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:46.288279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.288248 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:46.288381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.288359 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "kube-api-access-7t9vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:46.288431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.288393 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:46.288664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.288646 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo" pod "0f23de09-a8c2-44ce-99a2-f49ef3cb015a" (UID: "0f23de09-a8c2-44ce-99a2-f49ef3cb015a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:46.388467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388403 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388434 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388451 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9vj\" (UniqueName: \"kubernetes.io/projected/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-kube-api-access-7t9vj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388466 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388479 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388493 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.388931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.388506 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f23de09-a8c2-44ce-99a2-f49ef3cb015a-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.578463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.578433 4292 generic.go:356] "Generic (PLEG): container finished" podID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerID="4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545" exitCode=0 Oct 30 22:30:46.578651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.578479 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" Oct 30 22:30:46.578651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.578513 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerDied","Data":"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545"} Oct 30 22:30:46.578651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.578564 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd" event={"ID":"0f23de09-a8c2-44ce-99a2-f49ef3cb015a","Type":"ContainerDied","Data":"24361cda54324855bafd12fc85addc8b9936bfed4d52a04fbfd845959e50c9a6"} Oct 30 22:30:46.578651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.578617 4292 scope.go:122] "RemoveContainer" containerID="4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545" Oct 30 22:30:46.580104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.580085 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" Oct 30 22:30:46.580213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.580118 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8" event={"ID":"64b619ea-60fd-41de-9b26-bbd2c98de9a3","Type":"ContainerDied","Data":"ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312"} Oct 30 22:30:46.580213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.580172 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4865852b4b0374ba8da75d27b21b5ab937cd0485f1c4f409f51c65b3ff1312" Oct 30 22:30:46.589073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.589055 4292 scope.go:122] "RemoveContainer" containerID="13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963" Oct 30 22:30:46.598513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.598498 4292 scope.go:122] "RemoveContainer" containerID="4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545" Oct 30 22:30:46.598784 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:46.598764 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545\": container with ID starting with 4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545 not found: ID does not exist" containerID="4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545" Oct 30 22:30:46.598844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.598790 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545"} err="failed to get container status \"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545\": rpc error: code = NotFound desc = could not find container \"4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545\": container with ID starting with 4d837b213f5f8508ba8af870a37cf3ce0f329ea29bdede1fd77ef1a5eaad8545 not found: ID does not exist" Oct 30 22:30:46.598844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.598808 4292 scope.go:122] "RemoveContainer" containerID="13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963" Oct 30 22:30:46.599037 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:46.599014 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963\": container with ID starting with 13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963 not found: ID does not exist" containerID="13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963" Oct 30 22:30:46.599155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.599039 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963"} err="failed to get container status \"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963\": rpc error: code = NotFound desc = could not find container \"13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963\": container with ID starting with 13633bdd0186cff76a33b4235548c11e888c1cc3b46a8439d53afc987426c963 not found: ID does not exist" Oct 30 22:30:46.732884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.732858 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:30:46.752922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.752892 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:30:46.763052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.763026 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:30:46.782901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:46.782873 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afe802-openshift-default-54766d4cf7-x9chd"] Oct 30 22:30:48.149340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:30:48.149308 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" path="/var/lib/kubelet/pods/0f23de09-a8c2-44ce-99a2-f49ef3cb015a/volumes" Oct 30 22:30:56.641149 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:30:56.641113 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:31:01.557191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:01.557155 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44"] Oct 30 22:31:01.607074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:01.607040 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8e1227-ndh44"] Oct 30 22:31:01.753591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:01.753532 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" containerID="cri-o://a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46" gracePeriod=45 Oct 30 22:31:01.777073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:01.777033 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" containerID="cri-o://f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2" gracePeriod=45 Oct 30 22:31:02.051072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.051044 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:31:02.093668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.093633 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:31:02.100346 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100317 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100367 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100428 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100488 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100539 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.100741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100695 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\" (UniqueName: \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\") pod \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\" (UID: \"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407\") " Oct 30 22:31:02.100741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100710 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.100971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.100805 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.101259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.101234 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.101365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.101263 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.101365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.101279 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.105085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.103125 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "kube-api-access-bs2zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:02.105085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.103119 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:02.105085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.103354 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm" pod "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" (UID: "f2da699f-9b5f-41cb-9d5c-83c1aa2f4407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.148510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.147716 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cc9422-0eff-481c-9832-0e795a768aff" path="/var/lib/kubelet/pods/16cc9422-0eff-481c-9832-0e795a768aff/volumes" Oct 30 22:31:02.202281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202243 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.202482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202335 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\" (UniqueName: \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.202482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202434 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.202482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202464 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.202717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202517 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.202717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202526 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.202717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.202560 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\") pod \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\" (UID: \"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5\") " Oct 30 22:31:02.203138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.203383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203356 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.203951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203926 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.204047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203955 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.204047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203969 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.204047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203983 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.204047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.203999 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs2zz\" (UniqueName: \"kubernetes.io/projected/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407-kube-api-access-bs2zz\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.204047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.204015 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.205205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.205184 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:02.205793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.205758 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "kube-api-access-5lssp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:02.205899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.205873 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm" pod "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" (UID: "cb09c3d0-e9ac-4a5b-b80b-3d693df346b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:02.305131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.305092 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.305131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.305130 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.305335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.305145 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lssp\" (UniqueName: \"kubernetes.io/projected/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5-kube-api-access-5lssp\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:02.730684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.730651 4292 generic.go:356] "Generic (PLEG): container finished" podID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerID="f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2" exitCode=0 Oct 30 22:31:02.731165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.730698 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" Oct 30 22:31:02.731165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.730737 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" event={"ID":"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5","Type":"ContainerDied","Data":"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2"} Oct 30 22:31:02.731165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.730790 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4" event={"ID":"cb09c3d0-e9ac-4a5b-b80b-3d693df346b5","Type":"ContainerDied","Data":"ca7e345abf114d7d11119675371a2b546da709cce18eca52fdb217e43495a3f3"} Oct 30 22:31:02.731165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.730814 4292 scope.go:122] "RemoveContainer" containerID="f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2" Oct 30 22:31:02.732450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.732420 4292 generic.go:356] "Generic (PLEG): container finished" podID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerID="a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46" exitCode=0 Oct 30 22:31:02.732610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.732467 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" event={"ID":"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407","Type":"ContainerDied","Data":"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46"} Oct 30 22:31:02.732610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.732473 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" Oct 30 22:31:02.732610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.732494 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq" event={"ID":"f2da699f-9b5f-41cb-9d5c-83c1aa2f4407","Type":"ContainerDied","Data":"91b918704b7c38ca474c7392d40fd2a4103f8956de92768f09bfef6f787a4d42"} Oct 30 22:31:02.741212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.741192 4292 scope.go:122] "RemoveContainer" containerID="f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2" Oct 30 22:31:02.741484 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:02.741464 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2\": container with ID starting with f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2 not found: ID does not exist" containerID="f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2" Oct 30 22:31:02.741632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.741493 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2"} err="failed to get container status \"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2\": rpc error: code = NotFound desc = could not find container \"f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2\": container with ID starting with f7957c8926e9781ceead244feb5189580dd00d9f158ff51511c0da2718ca99b2 not found: ID does not exist" Oct 30 22:31:02.741632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.741518 4292 scope.go:122] "RemoveContainer" containerID="a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46" Oct 30 22:31:02.754374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.754148 4292 scope.go:122] "RemoveContainer" containerID="a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46" Oct 30 22:31:02.754699 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:02.754671 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46\": container with ID starting with a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46 not found: ID does not exist" containerID="a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46" Oct 30 22:31:02.754781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.754710 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46"} err="failed to get container status \"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46\": rpc error: code = NotFound desc = could not find container \"a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46\": container with ID starting with a13f8e4986cc4bc8ee7b56d74c4084090eebc4485220db9624ef3246393b9d46 not found: ID does not exist" Oct 30 22:31:02.903026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.902610 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:31:02.954553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:02.954520 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-2-w2-13b8d8-kserve-7f4cf9d7fd-j9mvq"] Oct 30 22:31:03.066206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:03.066171 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:31:03.072731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:03.072706 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-13b8d8/distinct-w2-13b8d8-kserve-88886576f-nwhc4"] Oct 30 22:31:04.149269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:04.149234 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" path="/var/lib/kubelet/pods/cb09c3d0-e9ac-4a5b-b80b-3d693df346b5/volumes" Oct 30 22:31:04.149829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:04.149711 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" path="/var/lib/kubelet/pods/f2da699f-9b5f-41cb-9d5c-83c1aa2f4407/volumes" Oct 30 22:31:10.954957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.954912 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2"] Oct 30 22:31:10.955490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955475 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955491 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955506 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955511 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955520 4292 cpu_manager.go:414] "containerMap: removing container" podUID="64b619ea-60fd-41de-9b26-bbd2c98de9a3" containerName="revoke-keys" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955526 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b619ea-60fd-41de-9b26-bbd2c98de9a3" containerName="revoke-keys" Oct 30 22:31:10.955534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955535 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955541 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955550 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955555 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955567 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955584 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ae0e55-b704-4446-a5f8-855033b855f0" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955595 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955599 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955609 4292 cpu_manager.go:414] "containerMap: removing container" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955614 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955630 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955634 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955717 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2da699f-9b5f-41cb-9d5c-83c1aa2f4407" containerName="main" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955727 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12303c3-00b0-4d84-b007-a3f040b44ea6" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955734 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d2a0e3-b6dc-417f-89da-4fd53c19b981" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955741 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8e9159-6c6e-4259-891c-d292138bf8d3" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955746 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b619ea-60fd-41de-9b26-bbd2c98de9a3" containerName="revoke-keys" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955753 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955758 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b99f5a-0442-454d-9e8e-a61459da0505" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955763 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d11fe00-6f4d-42ce-ba99-252a4b1f0dcc" containerName="cleanup" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955768 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f23de09-a8c2-44ce-99a2-f49ef3cb015a" containerName="istio-proxy" Oct 30 22:31:10.955766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955775 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28edfe07-b99a-43e5-b724-ac304d6de0ae" containerName="cleanup" Oct 30 22:31:10.956390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955783 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="530df160-469a-41ee-861c-c1e556fa2016" containerName="cleanup" Oct 30 22:31:10.956390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.955789 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb09c3d0-e9ac-4a5b-b80b-3d693df346b5" containerName="main" Oct 30 22:31:10.960543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:10.960524 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.021526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.021499 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2"] Oct 30 22:31:11.066946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.066901 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:11.067096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.067065 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:31:11.067174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.067091 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:11.067237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.067203 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:11.140276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.140243 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7gc\" (UniqueName: \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.140405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.140373 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.140475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.140457 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.242209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.242146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.242353 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.242332 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7gc\" (UniqueName: \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.242631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.242606 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.243196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.243178 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.244593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.244558 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.305465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.305438 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7gc\" (UniqueName: \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.573893 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.573863 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:11.954033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:11.954002 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2"] Oct 30 22:31:11.954406 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:11.954381 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88a5673_9173_4007_a966_90f6e20df6ec.slice/crio-20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790 WatchSource:0}: Error finding container 20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790: Status 404 returned error can't find the container with id 20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790 Oct 30 22:31:12.831968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:12.831930 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" event={"ID":"c88a5673-9173-4007-a966-90f6e20df6ec","Type":"ContainerStarted","Data":"7b20cef4c7120eb3cb726656594355a297f6807c00be8131125739e7a5765a6c"} Oct 30 22:31:12.832347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:12.831996 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" event={"ID":"c88a5673-9173-4007-a966-90f6e20df6ec","Type":"ContainerStarted","Data":"20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790"} Oct 30 22:31:12.952372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:12.952326 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" podStartSLOduration=2.9523127970000003 podStartE2EDuration="2.952312797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:12.95220396 +0000 UTC m=+2200.921023524" watchObservedRunningTime="2026-10-30 22:31:12.952312797 +0000 UTC m=+2200.921132361" Oct 30 22:31:13.840289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:13.840251 4292 generic.go:356] "Generic (PLEG): container finished" podID="c88a5673-9173-4007-a966-90f6e20df6ec" containerID="7b20cef4c7120eb3cb726656594355a297f6807c00be8131125739e7a5765a6c" exitCode=0 Oct 30 22:31:13.840718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:13.840339 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" event={"ID":"c88a5673-9173-4007-a966-90f6e20df6ec","Type":"ContainerDied","Data":"7b20cef4c7120eb3cb726656594355a297f6807c00be8131125739e7a5765a6c"} Oct 30 22:31:14.997650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:14.997623 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:15.085028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.084972 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\") pod \"c88a5673-9173-4007-a966-90f6e20df6ec\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " Oct 30 22:31:15.085132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.085074 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\") pod \"c88a5673-9173-4007-a966-90f6e20df6ec\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " Oct 30 22:31:15.085214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.085141 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\" (UniqueName: \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\") pod \"c88a5673-9173-4007-a966-90f6e20df6ec\" (UID: \"c88a5673-9173-4007-a966-90f6e20df6ec\") " Oct 30 22:31:15.085417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.085383 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca" pod "c88a5673-9173-4007-a966-90f6e20df6ec" (UID: "c88a5673-9173-4007-a966-90f6e20df6ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:15.085742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.085721 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:15.087112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.087092 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca" pod "c88a5673-9173-4007-a966-90f6e20df6ec" (UID: "c88a5673-9173-4007-a966-90f6e20df6ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:15.087174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.087092 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc" pod "c88a5673-9173-4007-a966-90f6e20df6ec" (UID: "c88a5673-9173-4007-a966-90f6e20df6ec"). InnerVolumeSpecName "kube-api-access-vd7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:15.187186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.187120 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7gc\" (UniqueName: \"kubernetes.io/projected/c88a5673-9173-4007-a966-90f6e20df6ec-kube-api-access-vd7gc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:15.187186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.187149 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88a5673-9173-4007-a966-90f6e20df6ec-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:15.854939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.854910 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" Oct 30 22:31:15.855144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.854949 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2" event={"ID":"c88a5673-9173-4007-a966-90f6e20df6ec","Type":"ContainerDied","Data":"20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790"} Oct 30 22:31:15.855144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:15.855000 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20120eb9a4a113d8da87f4d78499d036c4bac5d6a3a238efc1bebd7d839b5790" Oct 30 22:31:22.522349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:22.522315 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:31:22.634512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:22.634482 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:31:24.169372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:24.169340 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt"] Oct 30 22:31:24.175222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:24.175200 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-fz2qt"] Oct 30 22:31:26.172087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:26.172045 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea0748a-761d-43f2-a86b-42ac45a55146" path="/var/lib/kubelet/pods/6ea0748a-761d-43f2-a86b-42ac45a55146/volumes" Oct 30 22:31:26.641385 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:26.641344 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:31:27.523299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.523238 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" podUID="99e36320-98ed-4282-9a83-666aa7856d93" containerName="payload-processing" containerID="cri-o://ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411" gracePeriod=25 Oct 30 22:31:27.639622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.638657 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerName="payload-pre-processing" containerID="cri-o://f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74" gracePeriod=25 Oct 30 22:31:27.712825 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:27.712800 4292 logging.go:55] [core] [Channel #5354 SubChannel #5355]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.173:50052: connect: connection refused" Oct 30 22:31:27.830678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.830635 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:31:27.923802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.923772 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:31:27.959593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.958645 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\" (UniqueName: \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\") pod \"99e36320-98ed-4282-9a83-666aa7856d93\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " Oct 30 22:31:27.959593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.958758 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\") pod \"99e36320-98ed-4282-9a83-666aa7856d93\" (UID: \"99e36320-98ed-4282-9a83-666aa7856d93\") " Oct 30 22:31:27.960083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.960015 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume" pod "99e36320-98ed-4282-9a83-666aa7856d93" (UID: "99e36320-98ed-4282-9a83-666aa7856d93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:27.961914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.961886 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc" pod "99e36320-98ed-4282-9a83-666aa7856d93" (UID: "99e36320-98ed-4282-9a83-666aa7856d93"). InnerVolumeSpecName "kube-api-access-5lwdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:27.965068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.965043 4292 generic.go:356] "Generic (PLEG): container finished" podID="99e36320-98ed-4282-9a83-666aa7856d93" containerID="ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411" exitCode=0 Oct 30 22:31:27.965175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.965090 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" Oct 30 22:31:27.965175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.965123 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" event={"ID":"99e36320-98ed-4282-9a83-666aa7856d93","Type":"ContainerDied","Data":"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411"} Oct 30 22:31:27.965296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.965174 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk" event={"ID":"99e36320-98ed-4282-9a83-666aa7856d93","Type":"ContainerDied","Data":"446f0300f87a7ba39a7cb83c82f0c543fdaf28ff41eb6a3399eeacb0af785249"} Oct 30 22:31:27.965296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.965202 4292 scope.go:122] "RemoveContainer" containerID="ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411" Oct 30 22:31:27.967385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.967345 4292 generic.go:356] "Generic (PLEG): container finished" podID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerID="f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74" exitCode=0 Oct 30 22:31:27.967503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.967408 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" Oct 30 22:31:27.967503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.967431 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" event={"ID":"d06d7b86-75a3-4344-9b32-9bc1dc14fc22","Type":"ContainerDied","Data":"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74"} Oct 30 22:31:27.967503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.967477 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" event={"ID":"d06d7b86-75a3-4344-9b32-9bc1dc14fc22","Type":"ContainerDied","Data":"50da2b6fbbac9de94f45abd13502a8c87317b1dbdc6b5c47714472cdae043ed7"} Oct 30 22:31:27.977030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.977013 4292 scope.go:122] "RemoveContainer" containerID="ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411" Oct 30 22:31:27.977310 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:27.977281 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411\": container with ID starting with ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411 not found: ID does not exist" containerID="ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411" Oct 30 22:31:27.977382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.977321 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411"} err="failed to get container status \"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411\": rpc error: code = NotFound desc = could not find container \"ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411\": container with ID starting with ee37c35aac2c64a65a966db753174e0d16f08681571f314ae84d9c7aa2541411 not found: ID does not exist" Oct 30 22:31:27.977382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.977346 4292 scope.go:122] "RemoveContainer" containerID="f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74" Oct 30 22:31:27.988048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.988023 4292 scope.go:122] "RemoveContainer" containerID="f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74" Oct 30 22:31:27.988327 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:27.988306 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74\": container with ID starting with f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74 not found: ID does not exist" containerID="f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74" Oct 30 22:31:27.988415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:27.988334 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74"} err="failed to get container status \"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74\": rpc error: code = NotFound desc = could not find container \"f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74\": container with ID starting with f761b0456fb4a92031dc8f7d3c38a684032441b99e2c7e0c4a590a89b119bf74 not found: ID does not exist" Oct 30 22:31:28.085617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.068236 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\") pod \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " Oct 30 22:31:28.085617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.068372 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\" (UniqueName: \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\") pod \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\" (UID: \"d06d7b86-75a3-4344-9b32-9bc1dc14fc22\") " Oct 30 22:31:28.085617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.069073 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwdc\" (UniqueName: \"kubernetes.io/projected/99e36320-98ed-4282-9a83-666aa7856d93-kube-api-access-5lwdc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:28.085617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.069145 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e36320-98ed-4282-9a83-666aa7856d93-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:28.085617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.074844 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume" pod "d06d7b86-75a3-4344-9b32-9bc1dc14fc22" (UID: "d06d7b86-75a3-4344-9b32-9bc1dc14fc22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:28.131875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.131812 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q" pod "d06d7b86-75a3-4344-9b32-9bc1dc14fc22" (UID: "d06d7b86-75a3-4344-9b32-9bc1dc14fc22"). InnerVolumeSpecName "kube-api-access-gkz5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:28.171109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.171075 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:28.171236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.171112 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkz5q\" (UniqueName: \"kubernetes.io/projected/d06d7b86-75a3-4344-9b32-9bc1dc14fc22-kube-api-access-gkz5q\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:28.219484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.219454 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:31:28.224286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.224259 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7fd7ba-7d897bbf4f-n27pk"] Oct 30 22:31:28.388610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.388512 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:31:28.395208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.395176 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86"] Oct 30 22:31:28.713570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:28.713481 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7fd7ba-7d4846697f-97m86" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded" Oct 30 22:31:29.454298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:29.454262 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw"] Oct 30 22:31:29.504995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:29.504964 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-963c3a-252dw"] Oct 30 22:31:30.146183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:30.146153 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e36320-98ed-4282-9a83-666aa7856d93" path="/var/lib/kubelet/pods/99e36320-98ed-4282-9a83-666aa7856d93/volumes" Oct 30 22:31:30.146563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:30.146542 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" path="/var/lib/kubelet/pods/d06d7b86-75a3-4344-9b32-9bc1dc14fc22/volumes" Oct 30 22:31:30.146955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:30.146934 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6629d24-3d90-47b3-9b9b-9f4251cd2818" path="/var/lib/kubelet/pods/d6629d24-3d90-47b3-9b9b-9f4251cd2818/volumes" Oct 30 22:31:34.620323 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620282 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg"] Oct 30 22:31:34.620888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620872 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerName="payload-pre-processing" Oct 30 22:31:34.620933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620888 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerName="payload-pre-processing" Oct 30 22:31:34.620933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620901 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c88a5673-9173-4007-a966-90f6e20df6ec" containerName="revoke-keys" Oct 30 22:31:34.620933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620906 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88a5673-9173-4007-a966-90f6e20df6ec" containerName="revoke-keys" Oct 30 22:31:34.620933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620915 4292 cpu_manager.go:414] "containerMap: removing container" podUID="99e36320-98ed-4282-9a83-666aa7856d93" containerName="payload-processing" Oct 30 22:31:34.620933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.620920 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e36320-98ed-4282-9a83-666aa7856d93" containerName="payload-processing" Oct 30 22:31:34.621094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.621015 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06d7b86-75a3-4344-9b32-9bc1dc14fc22" containerName="payload-pre-processing" Oct 30 22:31:34.621094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.621022 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88a5673-9173-4007-a966-90f6e20df6ec" containerName="revoke-keys" Oct 30 22:31:34.621094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.621028 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e36320-98ed-4282-9a83-666aa7856d93" containerName="payload-processing" Oct 30 22:31:34.625544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.625526 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.692167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.691743 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg"] Oct 30 22:31:34.695291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.695262 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.695430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.695309 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wdv\" (UniqueName: \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.695507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.695459 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.734842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.734803 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:34.734842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.734803 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:34.734842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.734804 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:34.735136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.734884 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:31:34.796752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.796723 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.796937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.796836 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.796937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.796860 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wdv\" (UniqueName: \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.797522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.797495 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.799240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.799213 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.867488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.867455 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wdv\" (UniqueName: \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:34.939471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:34.939404 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:35.339961 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:35.339926 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d2e883_e7a0_487f_a078_eb55e661e7e7.slice/crio-4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68 WatchSource:0}: Error finding container 4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68: Status 404 returned error can't find the container with id 4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68 Oct 30 22:31:35.340222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:35.340189 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg"] Oct 30 22:31:36.025701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:36.025610 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" event={"ID":"10d2e883-e7a0-487f-a078-eb55e661e7e7","Type":"ContainerStarted","Data":"bd1ea81a7285341e52b47b403878b3b2ad136eb0f3dad4ff0e5300cee1b9cd21"} Oct 30 22:31:36.025701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:36.025667 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" event={"ID":"10d2e883-e7a0-487f-a078-eb55e661e7e7","Type":"ContainerStarted","Data":"4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68"} Oct 30 22:31:36.147392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:36.147341 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" podStartSLOduration=2.147322751 podStartE2EDuration="2.147322751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:36.144808804 +0000 UTC m=+2224.113628379" watchObservedRunningTime="2026-10-30 22:31:36.147322751 +0000 UTC m=+2224.116142315" Oct 30 22:31:36.331245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:36.331215 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:31:36.331433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:36.331408 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" containerID="cri-o://4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1" gracePeriod=30 Oct 30 22:31:37.034222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.034184 4292 generic.go:356] "Generic (PLEG): container finished" podID="10d2e883-e7a0-487f-a078-eb55e661e7e7" containerID="bd1ea81a7285341e52b47b403878b3b2ad136eb0f3dad4ff0e5300cee1b9cd21" exitCode=0 Oct 30 22:31:37.034695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.034267 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" event={"ID":"10d2e883-e7a0-487f-a078-eb55e661e7e7","Type":"ContainerDied","Data":"bd1ea81a7285341e52b47b403878b3b2ad136eb0f3dad4ff0e5300cee1b9cd21"} Oct 30 22:31:37.668133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.668098 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:31:37.668589 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.668541 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" containerID="cri-o://0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6" gracePeriod=30 Oct 30 22:31:37.900343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.900308 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:31:37.905976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.905954 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:37.947017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.946953 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:37.947017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.947004 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgl5k\" (UniqueName: \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:37.963226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.963200 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18515933\"" Oct 30 22:31:37.963481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.963466 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18515933-dockercfg-jmm68\"" Oct 30 22:31:37.971892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:37.971867 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:31:38.053598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.051206 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:38.053598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.051300 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgl5k\" (UniqueName: \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:38.053598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.052930 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:38.083327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.079734 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:31:38.086632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.086602 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.123089 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.123049 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:31:38.123753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.123726 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgl5k\" (UniqueName: \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\") pod \"payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:38.153290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.153256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8gw\" (UniqueName: \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.153475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.153322 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.219319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.219239 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:38.254529 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.254496 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8gw\" (UniqueName: \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.254690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.254562 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.255281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.255258 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.284219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.284194 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:38.318867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.318802 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8gw\" (UniqueName: \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\") pod \"payload-processing-e2e-mt-18515933-69cb874d4f-7plng\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.355236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.355210 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\") pod \"10d2e883-e7a0-487f-a078-eb55e661e7e7\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " Oct 30 22:31:38.355393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.355310 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\") pod \"10d2e883-e7a0-487f-a078-eb55e661e7e7\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " Oct 30 22:31:38.355393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.355351 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\" (UniqueName: \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\") pod \"10d2e883-e7a0-487f-a078-eb55e661e7e7\" (UID: \"10d2e883-e7a0-487f-a078-eb55e661e7e7\") " Oct 30 22:31:38.355639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.355609 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca" pod "10d2e883-e7a0-487f-a078-eb55e661e7e7" (UID: "10d2e883-e7a0-487f-a078-eb55e661e7e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:38.355914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.355889 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.357371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.357348 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv" pod "10d2e883-e7a0-487f-a078-eb55e661e7e7" (UID: "10d2e883-e7a0-487f-a078-eb55e661e7e7"). InnerVolumeSpecName "kube-api-access-d4wdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.357449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.357370 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca" pod "10d2e883-e7a0-487f-a078-eb55e661e7e7" (UID: "10d2e883-e7a0-487f-a078-eb55e661e7e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:38.422770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.422744 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:38.457062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.457023 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d2e883-e7a0-487f-a078-eb55e661e7e7-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.457062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.457049 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wdv\" (UniqueName: \"kubernetes.io/projected/10d2e883-e7a0-487f-a078-eb55e661e7e7-kube-api-access-d4wdv\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.599430 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:38.599399 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27599305_8e1a_452d_afb3_dd9a446ed3a8.slice/crio-7157b6adcc029d7bfd03f600e1b3acdee827ec4571bdfb94e0b4c66f53e4156b WatchSource:0}: Error finding container 7157b6adcc029d7bfd03f600e1b3acdee827ec4571bdfb94e0b4c66f53e4156b: Status 404 returned error can't find the container with id 7157b6adcc029d7bfd03f600e1b3acdee827ec4571bdfb94e0b4c66f53e4156b Oct 30 22:31:38.600343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.600316 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:31:38.801541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.801515 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:31:38.802608 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:38.802568 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0562400_f4ae_4961_9ee0_4670d5a16e90.slice/crio-a16e5a1342e4f17fff54fb56816856d00b704b07c3d10e1ac114c88a3c71a76a WatchSource:0}: Error finding container a16e5a1342e4f17fff54fb56816856d00b704b07c3d10e1ac114c88a3c71a76a: Status 404 returned error can't find the container with id a16e5a1342e4f17fff54fb56816856d00b704b07c3d10e1ac114c88a3c71a76a Oct 30 22:31:38.848102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.848072 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq"] Oct 30 22:31:38.848866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.848835 4292 cpu_manager.go:414] "containerMap: removing container" podUID="10d2e883-e7a0-487f-a078-eb55e661e7e7" containerName="revoke-keys" Oct 30 22:31:38.848866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.848859 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d2e883-e7a0-487f-a078-eb55e661e7e7" containerName="revoke-keys" Oct 30 22:31:38.849233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.849049 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d2e883-e7a0-487f-a078-eb55e661e7e7" containerName="revoke-keys" Oct 30 22:31:38.853820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.853802 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.862131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.862101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4pv\" (UniqueName: \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.862267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.862224 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.862341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.862265 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.915295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.915266 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq"] Oct 30 22:31:38.964653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.963564 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4pv\" (UniqueName: \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.964653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.963643 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.964653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.963677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.964653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.964552 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.966770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.966738 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:38.985265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.985240 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:31:38.985514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:38.985489 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" containerID="cri-o://486543175baf4084f7d1e33d3ca2b0037571608dca8d51e14d44b14dcf485acf" gracePeriod=30 Oct 30 22:31:39.027454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.027425 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4pv\" (UniqueName: \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:39.068142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.068058 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" event={"ID":"10d2e883-e7a0-487f-a078-eb55e661e7e7","Type":"ContainerDied","Data":"4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68"} Oct 30 22:31:39.068142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.068134 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e792ac33564683a7bbe721d85983ed0cac6727480f29ae3685836c3eef39e68" Oct 30 22:31:39.068555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.068209 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg" Oct 30 22:31:39.132643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.128169 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" event={"ID":"27599305-8e1a-452d-afb3-dd9a446ed3a8","Type":"ContainerStarted","Data":"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef"} Oct 30 22:31:39.132643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.128237 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" event={"ID":"27599305-8e1a-452d-afb3-dd9a446ed3a8","Type":"ContainerStarted","Data":"7157b6adcc029d7bfd03f600e1b3acdee827ec4571bdfb94e0b4c66f53e4156b"} Oct 30 22:31:39.132643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.128268 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:39.147601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.140256 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" event={"ID":"f0562400-f4ae-4961-9ee0-4670d5a16e90","Type":"ContainerStarted","Data":"3647d5257689e9129433e8f6dfe99e7ec78837c1fc12a871e928bee9c37b9c1e"} Oct 30 22:31:39.147601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.140308 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" event={"ID":"f0562400-f4ae-4961-9ee0-4670d5a16e90","Type":"ContainerStarted","Data":"a16e5a1342e4f17fff54fb56816856d00b704b07c3d10e1ac114c88a3c71a76a"} Oct 30 22:31:39.147601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.140941 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:39.179335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.179300 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:39.427650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.427510 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" podStartSLOduration=2.427493942 podStartE2EDuration="2.427493942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:39.251974729 +0000 UTC m=+2227.220794295" watchObservedRunningTime="2026-10-30 22:31:39.427493942 +0000 UTC m=+2227.396313519" Oct 30 22:31:39.428695 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.428659 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" podStartSLOduration=1.428646782 podStartE2EDuration="1.428646782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:39.425521946 +0000 UTC m=+2227.394341511" watchObservedRunningTime="2026-10-30 22:31:39.428646782 +0000 UTC m=+2227.397466346" Oct 30 22:31:39.568049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:39.568010 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq"] Oct 30 22:31:39.568776 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:39.568746 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011d6f96_99bf_4a35_aa53_75d438f0a58e.slice/crio-2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d WatchSource:0}: Error finding container 2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d: Status 404 returned error can't find the container with id 2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d Oct 30 22:31:40.152828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.152736 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" event={"ID":"011d6f96-99bf-4a35-aa53-75d438f0a58e","Type":"ContainerStarted","Data":"599b8f1b070bc21990e563eeb98010c4ad637b05f8ba1ea1130d7cbd2a89f62b"} Oct 30 22:31:40.152828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.152791 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" event={"ID":"011d6f96-99bf-4a35-aa53-75d438f0a58e","Type":"ContainerStarted","Data":"2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d"} Oct 30 22:31:40.277220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.277159 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" podStartSLOduration=2.277144049 podStartE2EDuration="2.277144049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:40.276375519 +0000 UTC m=+2228.245195084" watchObservedRunningTime="2026-10-30 22:31:40.277144049 +0000 UTC m=+2228.245963654" Oct 30 22:31:40.590251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.590218 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:31:40.594240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.594220 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.655543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.655518 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cd5407-openshift-default-dockercfg-7mj7l\"" Oct 30 22:31:40.685883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.685852 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.685894 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.685996 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686177 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686283 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdzx\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686354 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686393 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.686464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.686452 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.708616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.708566 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:31:40.788502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788455 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.788730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788542 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.788730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788660 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.788883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788747 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdzx\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.788883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788807 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.788883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788855 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.789161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788908 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.789161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.788987 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.789161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.789014 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.789837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.789803 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.789959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.789822 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.790110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.790067 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.791536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.790307 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.791536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.790750 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.797116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.796446 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.799188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.799137 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.854624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.854546 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.854775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.854695 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdzx\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\") pod \"e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:40.941040 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:40.941004 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:41.161466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.161386 4292 generic.go:356] "Generic (PLEG): container finished" podID="011d6f96-99bf-4a35-aa53-75d438f0a58e" containerID="599b8f1b070bc21990e563eeb98010c4ad637b05f8ba1ea1130d7cbd2a89f62b" exitCode=0 Oct 30 22:31:41.161876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.161478 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" event={"ID":"011d6f96-99bf-4a35-aa53-75d438f0a58e","Type":"ContainerDied","Data":"599b8f1b070bc21990e563eeb98010c4ad637b05f8ba1ea1130d7cbd2a89f62b"} Oct 30 22:31:41.422564 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:41.422535 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c5d61c_1f0a_406f_aae6_0d9e4841ce2a.slice/crio-cbd1625d0c6736e23a7196940f1f63f0712c91447f69df62f68a9db34ead62a9 WatchSource:0}: Error finding container cbd1625d0c6736e23a7196940f1f63f0712c91447f69df62f68a9db34ead62a9: Status 404 returned error can't find the container with id cbd1625d0c6736e23a7196940f1f63f0712c91447f69df62f68a9db34ead62a9 Oct 30 22:31:41.422691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.422667 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:31:41.437945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.437914 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:41.438061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.437988 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:41.438061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.438031 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:41.633016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.632994 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:31:41.701847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.701770 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.701847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.701817 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.701891 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.701948 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.701978 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702020 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702067 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.702255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702087 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.702255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702196 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.702255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702218 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702273 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\") pod \"5ce900df-26aa-4630-8bca-bb946483a843\" (UID: \"5ce900df-26aa-4630-8bca-bb946483a843\") " Oct 30 22:31:41.702716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702693 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.702800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702738 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.702800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702759 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.702800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702774 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.703013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.702991 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:41.704473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.704451 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:41.704601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.704493 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.704800 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.704777 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "kube-api-access-klp9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:41.704996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.704978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo" pod "5ce900df-26aa-4630-8bca-bb946483a843" (UID: "5ce900df-26aa-4630-8bca-bb946483a843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:41.804243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804211 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.804243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804239 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce900df-26aa-4630-8bca-bb946483a843-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.804243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804248 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.804478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804258 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce900df-26aa-4630-8bca-bb946483a843-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.804478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804267 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klp9d\" (UniqueName: \"kubernetes.io/projected/5ce900df-26aa-4630-8bca-bb946483a843-kube-api-access-klp9d\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.804478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:41.804275 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce900df-26aa-4630-8bca-bb946483a843-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.168829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.168801 4292 generic.go:356] "Generic (PLEG): container finished" podID="5ce900df-26aa-4630-8bca-bb946483a843" containerID="4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1" exitCode=0 Oct 30 22:31:42.169177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.168843 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" Oct 30 22:31:42.169177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.168880 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerDied","Data":"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1"} Oct 30 22:31:42.169177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.168927 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr" event={"ID":"5ce900df-26aa-4630-8bca-bb946483a843","Type":"ContainerDied","Data":"331c35c4e76e423f206da97e819bfef4f3242ea738609b69f33b87ada0923d7f"} Oct 30 22:31:42.169177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.168950 4292 scope.go:122] "RemoveContainer" containerID="4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1" Oct 30 22:31:42.170951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.170919 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerStarted","Data":"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d"} Oct 30 22:31:42.171120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.170978 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerStarted","Data":"cbd1625d0c6736e23a7196940f1f63f0712c91447f69df62f68a9db34ead62a9"} Oct 30 22:31:42.180118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.180100 4292 scope.go:122] "RemoveContainer" containerID="6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389" Oct 30 22:31:42.190213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.190189 4292 scope.go:122] "RemoveContainer" containerID="4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1" Oct 30 22:31:42.190518 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:42.190490 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1\": container with ID starting with 4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1 not found: ID does not exist" containerID="4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1" Oct 30 22:31:42.190605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.190523 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1"} err="failed to get container status \"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1\": rpc error: code = NotFound desc = could not find container \"4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1\": container with ID starting with 4eb33b18268191b64d5d6d137afdc27105ad99076b84dabcf872f2e3e98a35b1 not found: ID does not exist" Oct 30 22:31:42.190605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.190540 4292 scope.go:122] "RemoveContainer" containerID="6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389" Oct 30 22:31:42.190872 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:42.190856 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389\": container with ID starting with 6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389 not found: ID does not exist" containerID="6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389" Oct 30 22:31:42.190931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.190879 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389"} err="failed to get container status \"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389\": rpc error: code = NotFound desc = could not find container \"6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389\": container with ID starting with 6d5aaaa6ef3f72e85898845eadc8c28fce98ee802b2e50bc5285787c59e5a389 not found: ID does not exist" Oct 30 22:31:42.303638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.303606 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:42.322187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.322145 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podStartSLOduration=2.322133517 podStartE2EDuration="2.322133517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:42.320447723 +0000 UTC m=+2230.289267286" watchObservedRunningTime="2026-10-30 22:31:42.322133517 +0000 UTC m=+2230.290953080" Oct 30 22:31:42.412222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.412160 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\" (UniqueName: \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\") pod \"011d6f96-99bf-4a35-aa53-75d438f0a58e\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " Oct 30 22:31:42.412222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.412199 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\") pod \"011d6f96-99bf-4a35-aa53-75d438f0a58e\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " Oct 30 22:31:42.412374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.412305 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\") pod \"011d6f96-99bf-4a35-aa53-75d438f0a58e\" (UID: \"011d6f96-99bf-4a35-aa53-75d438f0a58e\") " Oct 30 22:31:42.412674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.412647 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca" pod "011d6f96-99bf-4a35-aa53-75d438f0a58e" (UID: "011d6f96-99bf-4a35-aa53-75d438f0a58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:42.412878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.412861 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.414321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.414297 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca" pod "011d6f96-99bf-4a35-aa53-75d438f0a58e" (UID: "011d6f96-99bf-4a35-aa53-75d438f0a58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:42.414441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.414423 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv" pod "011d6f96-99bf-4a35-aa53-75d438f0a58e" (UID: "011d6f96-99bf-4a35-aa53-75d438f0a58e"). InnerVolumeSpecName "kube-api-access-vv4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:42.457994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.457943 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:31:42.461904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.461873 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-56c628-openshift-default-5d69bbc45-6jmlr"] Oct 30 22:31:42.514374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.514292 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4pv\" (UniqueName: \"kubernetes.io/projected/011d6f96-99bf-4a35-aa53-75d438f0a58e-kube-api-access-vv4pv\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.514374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.514377 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011d6f96-99bf-4a35-aa53-75d438f0a58e-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.931071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.931046 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:31:42.941539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.941488 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:42.947156 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:42.947133 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:43.019938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.019906 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020002 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020039 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020058 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020079 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020136 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020225 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020269 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020348 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.020413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020363 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:43.020413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020388 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\") pod \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\" (UID: \"2754a886-2f9f-4515-9896-1fb4cbc58cb4\") " Oct 30 22:31:43.020635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020368 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.020635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020450 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.020635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020466 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.020877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020857 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.020877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020878 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.021004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020888 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.021004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020897 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.021004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.020909 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.022505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.022479 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:43.022625 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.022539 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.022805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.022789 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:43.022866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.022811 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps" pod "2754a886-2f9f-4515-9896-1fb4cbc58cb4" (UID: "2754a886-2f9f-4515-9896-1fb4cbc58cb4"). InnerVolumeSpecName "kube-api-access-5l2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:43.122959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.122913 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.122959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.122958 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.123149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.122975 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2ps\" (UniqueName: \"kubernetes.io/projected/2754a886-2f9f-4515-9896-1fb4cbc58cb4-kube-api-access-5l2ps\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.123149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.122993 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754a886-2f9f-4515-9896-1fb4cbc58cb4-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.178235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.178204 4292 generic.go:356] "Generic (PLEG): container finished" podID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerID="0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6" exitCode=0 Oct 30 22:31:43.178657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.178263 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" Oct 30 22:31:43.178657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.178287 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerDied","Data":"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6"} Oct 30 22:31:43.178657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.178341 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt" event={"ID":"2754a886-2f9f-4515-9896-1fb4cbc58cb4","Type":"ContainerDied","Data":"e1f08955878e50e5b40b68bdf0768d5f4dae83231587e679862258acd0c8a702"} Oct 30 22:31:43.178657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.178368 4292 scope.go:122] "RemoveContainer" containerID="0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6" Oct 30 22:31:43.179718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.179688 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" Oct 30 22:31:43.179848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.179716 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq" event={"ID":"011d6f96-99bf-4a35-aa53-75d438f0a58e","Type":"ContainerDied","Data":"2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d"} Oct 30 22:31:43.179848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.179756 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5bf5c1ac45ae9a751af66468ed3fe17ea9f75931d63569a63abd5a13d8d10d" Oct 30 22:31:43.181320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.181298 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:43.182470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.182452 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:31:43.193160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.193132 4292 scope.go:122] "RemoveContainer" containerID="9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742" Oct 30 22:31:43.203445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.203428 4292 scope.go:122] "RemoveContainer" containerID="0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6" Oct 30 22:31:43.203749 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:43.203730 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6\": container with ID starting with 0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6 not found: ID does not exist" containerID="0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6" Oct 30 22:31:43.203889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.203761 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6"} err="failed to get container status \"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6\": rpc error: code = NotFound desc = could not find container \"0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6\": container with ID starting with 0c7b04f100739de9f94eb5160c6c1086c45d0ac2e97824828098bbc6f81945f6 not found: ID does not exist" Oct 30 22:31:43.203889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.203785 4292 scope.go:122] "RemoveContainer" containerID="9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742" Oct 30 22:31:43.204064 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:43.204045 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742\": container with ID starting with 9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742 not found: ID does not exist" containerID="9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742" Oct 30 22:31:43.204107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.204070 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742"} err="failed to get container status \"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742\": rpc error: code = NotFound desc = could not find container \"9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742\": container with ID starting with 9541600ef3a3611111421d3532629ec6bbf10183a9c3ac3d9cebea9789a50742 not found: ID does not exist" Oct 30 22:31:43.263724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.263704 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:31:43.312003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.311977 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:31:43.319338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:43.319310 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6d4d3c-openshift-default-59cfc69bd5-99dzt"] Oct 30 22:31:44.146990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.146958 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" path="/var/lib/kubelet/pods/2754a886-2f9f-4515-9896-1fb4cbc58cb4/volumes" Oct 30 22:31:44.147450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.147427 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce900df-26aa-4630-8bca-bb946483a843" path="/var/lib/kubelet/pods/5ce900df-26aa-4630-8bca-bb946483a843/volumes" Oct 30 22:31:44.195163 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.195129 4292 generic.go:356] "Generic (PLEG): container finished" podID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerID="486543175baf4084f7d1e33d3ca2b0037571608dca8d51e14d44b14dcf485acf" exitCode=0 Oct 30 22:31:44.195610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.195200 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerDied","Data":"486543175baf4084f7d1e33d3ca2b0037571608dca8d51e14d44b14dcf485acf"} Oct 30 22:31:44.195610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.195273 4292 scope.go:122] "RemoveContainer" containerID="f3e58cd5021bca2f876a05a1835a4f348862e4620be7bb4857ff3178fb147e33" Oct 30 22:31:44.257477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.257428 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:31:44.336525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336497 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336688 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336731 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336805 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.336901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336890 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.337080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.336966 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.337080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337007 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\") pod \"880c84c7-7c67-4501-9f05-f3256c11d4e1\" (UID: \"880c84c7-7c67-4501-9f05-f3256c11d4e1\") " Oct 30 22:31:44.337209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:44.337209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337139 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.337316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337217 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.337378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337328 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.337439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337424 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.337658 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337635 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c84c7-7c67-4501-9f05-f3256c11d4e1-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.337761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337661 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.337761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337675 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.337761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337688 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.337761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.337702 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.339099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.339080 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.339173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.339143 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.339310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.339291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "kube-api-access-5lp8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.339480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.339464 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy" pod "880c84c7-7c67-4501-9f05-f3256c11d4e1" (UID: "880c84c7-7c67-4501-9f05-f3256c11d4e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.438841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.438809 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.438841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.438837 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.439047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.438847 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lp8g\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-kube-api-access-5lp8g\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.439047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:44.438857 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c84c7-7c67-4501-9f05-f3256c11d4e1-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.204221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:45.204192 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" Oct 30 22:31:45.204696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:45.204235 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt" event={"ID":"880c84c7-7c67-4501-9f05-f3256c11d4e1","Type":"ContainerDied","Data":"b62d259ef7e25f6eb6c76c7509d48bc452d34b91503adb50bd73077dd64b1bd2"} Oct 30 22:31:45.204696 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:45.204295 4292 scope.go:122] "RemoveContainer" containerID="486543175baf4084f7d1e33d3ca2b0037571608dca8d51e14d44b14dcf485acf" Oct 30 22:31:45.374184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:45.374151 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:31:45.424540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:45.424505 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-963c3a-openshift-default-8595756565-tlrnt"] Oct 30 22:31:46.147239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:46.147207 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" path="/var/lib/kubelet/pods/880c84c7-7c67-4501-9f05-f3256c11d4e1/volumes" Oct 30 22:31:50.168021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.167891 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:31:50.436310 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436233 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:31:50.436808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436795 4292 cpu_manager.go:414] "containerMap: removing container" podUID="011d6f96-99bf-4a35-aa53-75d438f0a58e" containerName="revoke-keys" Oct 30 22:31:50.436852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436809 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011d6f96-99bf-4a35-aa53-75d438f0a58e" containerName="revoke-keys" Oct 30 22:31:50.436852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436829 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.436852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436835 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.436852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436847 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436853 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436861 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436866 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436879 4292 cpu_manager.go:414] "containerMap: removing container" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436883 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436899 4292 cpu_manager.go:414] "containerMap: removing container" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.436966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436904 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436989 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.436999 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="880c84c7-7c67-4501-9f05-f3256c11d4e1" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.437007 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.437015 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="011d6f96-99bf-4a35-aa53-75d438f0a58e" containerName="revoke-keys" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.437023 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.437030 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.437176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.437035 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce900df-26aa-4630-8bca-bb946483a843" containerName="istio-proxy" Oct 30 22:31:50.441889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.441872 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.500861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.500834 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cd5407-dockercfg-ctpd7\"" Oct 30 22:31:50.500861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.500844 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cd5407\"" Oct 30 22:31:50.505008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.504987 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:31:50.542528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.542475 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm79\" (UniqueName: \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.543623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.543592 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.613280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.609684 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:31:50.613280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.611052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.613280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.611078 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2754a886-2f9f-4515-9896-1fb4cbc58cb4" containerName="istio-proxy" Oct 30 22:31:50.615895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.615859 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.645485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.645456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.645677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.645664 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm79\" (UniqueName: \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.646135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.646113 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.662332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.662306 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:31:50.711964 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.711910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm79\" (UniqueName: \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\") pod \"payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.746857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.746824 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9b74\" (UniqueName: \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.746989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.746966 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.755507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.755480 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:50.847869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.847835 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.848014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.847977 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9b74\" (UniqueName: \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.848523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.848501 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.911672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.911635 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9b74\" (UniqueName: \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\") pod \"payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:50.932435 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:50.932412 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:51.138157 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:51.138127 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636da51d_84f5_4495_a50f_f2c3b78d5680.slice/crio-72f10fc890195c2a22e9579701672e119dbedd22619f776f25c7d6895e421720 WatchSource:0}: Error finding container 72f10fc890195c2a22e9579701672e119dbedd22619f776f25c7d6895e421720: Status 404 returned error can't find the container with id 72f10fc890195c2a22e9579701672e119dbedd22619f776f25c7d6895e421720 Oct 30 22:31:51.138379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.138350 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:31:51.165038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.165018 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:31:51.225179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.225145 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:31:51.245272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.245211 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" event={"ID":"636da51d-84f5-4495-a50f-f2c3b78d5680","Type":"ContainerStarted","Data":"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da"} Oct 30 22:31:51.245272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.245261 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" event={"ID":"636da51d-84f5-4495-a50f-f2c3b78d5680","Type":"ContainerStarted","Data":"72f10fc890195c2a22e9579701672e119dbedd22619f776f25c7d6895e421720"} Oct 30 22:31:51.245407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.245298 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:31:51.317483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.317453 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:31:51.321740 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:51.321711 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8bfe25_3e44_4209_ae79_c5dbc179fd25.slice/crio-7552a82bc2260adb89476c122841ea8692a88e282c2972d5120c7580405994da WatchSource:0}: Error finding container 7552a82bc2260adb89476c122841ea8692a88e282c2972d5120c7580405994da: Status 404 returned error can't find the container with id 7552a82bc2260adb89476c122841ea8692a88e282c2972d5120c7580405994da Oct 30 22:31:51.533451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:51.533343 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" podStartSLOduration=1.5333247509999999 podStartE2EDuration="1.533324751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:51.533206234 +0000 UTC m=+2239.502025799" watchObservedRunningTime="2026-10-30 22:31:51.533324751 +0000 UTC m=+2239.502144317" Oct 30 22:31:52.253347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:52.253308 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" event={"ID":"2c8bfe25-3e44-4209-ae79-c5dbc179fd25","Type":"ContainerStarted","Data":"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf"} Oct 30 22:31:52.253777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:52.253369 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" event={"ID":"2c8bfe25-3e44-4209-ae79-c5dbc179fd25","Type":"ContainerStarted","Data":"7552a82bc2260adb89476c122841ea8692a88e282c2972d5120c7580405994da"} Oct 30 22:31:52.253777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:52.253462 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:31:52.374639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:52.374589 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" podStartSLOduration=2.374560277 podStartE2EDuration="2.374560277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:52.372935495 +0000 UTC m=+2240.341755060" watchObservedRunningTime="2026-10-30 22:31:52.374560277 +0000 UTC m=+2240.343379880" Oct 30 22:31:56.130023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.129983 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6"] Oct 30 22:31:56.134327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.134308 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.147138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.147105 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.147388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.147365 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlnh\" (UniqueName: \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.147507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.147488 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.194898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.194865 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl"] Oct 30 22:31:56.199610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.199589 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.225977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.225934 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" containerID="cri-o://7488f2acc2261f0533b9059774232f857448a9c827cd41b32e435f3ecb3450d2" gracePeriod=25 Oct 30 22:31:56.241289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.241223 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:56.241289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.241256 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:56.241289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.241268 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:56.241508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.241412 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:31:56.244845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.244820 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6"] Oct 30 22:31:56.248666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248640 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlnh\" (UniqueName: \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.248776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248724 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.248845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248772 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.248845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248811 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.248954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248842 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4xh7\" (UniqueName: \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.249024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.248999 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.249472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.249451 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.250945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.250927 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.259715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.259692 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl"] Oct 30 22:31:56.323007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.322975 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlnh\" (UniqueName: \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\") pod \"maas-api-revoke-sub-69069d29-drxd6\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.350221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.350195 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.350467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.350448 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.350518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.350502 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4xh7\" (UniqueName: \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.350917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.350896 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.352509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.352491 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.414553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.414503 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4xh7\" (UniqueName: \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\") pod \"maas-api-revoke-sub-8c4f6954-zkhbl\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.451320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.451294 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:56.514511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.514250 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:56.645492 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:56.645407 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:31:56.828532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.828492 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6"] Oct 30 22:31:56.828796 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:56.828770 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a9868_ed11_4ee4_9ce6_ad98ea2e6a5d.slice/crio-decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e WatchSource:0}: Error finding container decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e: Status 404 returned error can't find the container with id decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e Oct 30 22:31:56.941326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:56.941302 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl"] Oct 30 22:31:56.941464 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:31:56.941444 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0bfc7d_7252_4559_a66e_38d50da85f68.slice/crio-2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16 WatchSource:0}: Error finding container 2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16: Status 404 returned error can't find the container with id 2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16 Oct 30 22:31:57.291000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:57.290961 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" event={"ID":"7a0bfc7d-7252-4559-a66e-38d50da85f68","Type":"ContainerStarted","Data":"9b507c0e2533a14ee3e57fd372c6fad36ab9f8362bd7c3291b2a98b669ce4f9a"} Oct 30 22:31:57.291402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:57.291022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" event={"ID":"7a0bfc7d-7252-4559-a66e-38d50da85f68","Type":"ContainerStarted","Data":"2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16"} Oct 30 22:31:57.292807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:57.292784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" event={"ID":"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d","Type":"ContainerStarted","Data":"05799103e2c90f71574287338309fc41e61eff6545693d33801b7bc498841001"} Oct 30 22:31:57.292926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:57.292829 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" event={"ID":"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d","Type":"ContainerStarted","Data":"decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e"} Oct 30 22:31:57.411633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:57.411536 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" podStartSLOduration=1.411524717 podStartE2EDuration="1.411524717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:57.411031932 +0000 UTC m=+2245.379851497" watchObservedRunningTime="2026-10-30 22:31:57.411524717 +0000 UTC m=+2245.380344258" Oct 30 22:31:58.284196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.284151 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" containerID="cri-o://555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376" gracePeriod=45 Oct 30 22:31:58.301351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.301308 4292 generic.go:356] "Generic (PLEG): container finished" podID="7a0bfc7d-7252-4559-a66e-38d50da85f68" containerID="9b507c0e2533a14ee3e57fd372c6fad36ab9f8362bd7c3291b2a98b669ce4f9a" exitCode=0 Oct 30 22:31:58.301701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.301401 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" event={"ID":"7a0bfc7d-7252-4559-a66e-38d50da85f68","Type":"ContainerDied","Data":"9b507c0e2533a14ee3e57fd372c6fad36ab9f8362bd7c3291b2a98b669ce4f9a"} Oct 30 22:31:58.303295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.303270 4292 generic.go:356] "Generic (PLEG): container finished" podID="c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" containerID="05799103e2c90f71574287338309fc41e61eff6545693d33801b7bc498841001" exitCode=0 Oct 30 22:31:58.303403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.303346 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" event={"ID":"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d","Type":"ContainerDied","Data":"05799103e2c90f71574287338309fc41e61eff6545693d33801b7bc498841001"} Oct 30 22:31:58.386729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.386699 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Oct 30 22:31:58.427938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.427889 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" podStartSLOduration=2.427875592 podStartE2EDuration="2.427875592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:57.554240327 +0000 UTC m=+2245.523059892" watchObservedRunningTime="2026-10-30 22:31:58.427875592 +0000 UTC m=+2246.396695156" Oct 30 22:31:58.567339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.567309 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:31:58.679311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679282 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\" (UniqueName: \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679373 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679426 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679709 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\") pod \"690fa1e2-df19-4660-a8ae-f476f18a61f7\" (UID: \"690fa1e2-df19-4660-a8ae-f476f18a61f7\") " Oct 30 22:31:58.679805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679729 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:58.679841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:58.679957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.679937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:58.680209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.680184 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:58.680342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.680227 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:58.680342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.680242 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:58.681913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.681892 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:58.682012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.681997 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:58.682085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.682064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s" pod "690fa1e2-df19-4660-a8ae-f476f18a61f7" (UID: "690fa1e2-df19-4660-a8ae-f476f18a61f7"). InnerVolumeSpecName "kube-api-access-mwf8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:58.781462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.781430 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690fa1e2-df19-4660-a8ae-f476f18a61f7-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:58.781462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.781460 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwf8s\" (UniqueName: \"kubernetes.io/projected/690fa1e2-df19-4660-a8ae-f476f18a61f7-kube-api-access-mwf8s\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:58.781694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:58.781470 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690fa1e2-df19-4660-a8ae-f476f18a61f7-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.220652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.220608 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:59.311687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.311644 4292 generic.go:356] "Generic (PLEG): container finished" podID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerID="555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376" exitCode=0 Oct 30 22:31:59.311687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.311685 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" Oct 30 22:31:59.312170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.311736 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" event={"ID":"690fa1e2-df19-4660-a8ae-f476f18a61f7","Type":"ContainerDied","Data":"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376"} Oct 30 22:31:59.312170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.311784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf" event={"ID":"690fa1e2-df19-4660-a8ae-f476f18a61f7","Type":"ContainerDied","Data":"8bce7e67172bbcec138574a97ddf35d114f5f3c9e9aef990df421259fb3772e9"} Oct 30 22:31:59.312170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.311804 4292 scope.go:122] "RemoveContainer" containerID="555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376" Oct 30 22:31:59.324167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.324143 4292 scope.go:122] "RemoveContainer" containerID="555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376" Oct 30 22:31:59.324486 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:31:59.324466 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376\": container with ID starting with 555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376 not found: ID does not exist" containerID="555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376" Oct 30 22:31:59.324545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.324494 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376"} err="failed to get container status \"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376\": rpc error: code = NotFound desc = could not find container \"555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376\": container with ID starting with 555b6e5cdb8c2a486da46f67228ee6e77f6ccfd31921b7735a2c56dfefa6d376 not found: ID does not exist" Oct 30 22:31:59.464878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.464853 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:31:59.482823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.482796 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:31:59.489498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.489475 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\" (UniqueName: \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\") pod \"7a0bfc7d-7252-4559-a66e-38d50da85f68\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " Oct 30 22:31:59.489701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.489685 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\") pod \"7a0bfc7d-7252-4559-a66e-38d50da85f68\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " Oct 30 22:31:59.489782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.489750 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\") pod \"7a0bfc7d-7252-4559-a66e-38d50da85f68\" (UID: \"7a0bfc7d-7252-4559-a66e-38d50da85f68\") " Oct 30 22:31:59.490022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.489998 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca" pod "7a0bfc7d-7252-4559-a66e-38d50da85f68" (UID: "7a0bfc7d-7252-4559-a66e-38d50da85f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:59.490263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.490243 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.491601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.491569 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7" pod "7a0bfc7d-7252-4559-a66e-38d50da85f68" (UID: "7a0bfc7d-7252-4559-a66e-38d50da85f68"). InnerVolumeSpecName "kube-api-access-h4xh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:59.492052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.492032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca" pod "7a0bfc7d-7252-4559-a66e-38d50da85f68" (UID: "7a0bfc7d-7252-4559-a66e-38d50da85f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:59.504992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.504973 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:31:59.532828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.532807 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-77cb1f/distinct-w1-77cb1f-kserve-5996685c7b-nmqqf"] Oct 30 22:31:59.591466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.591442 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\") pod \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " Oct 30 22:31:59.591567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.591479 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\") pod \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " Oct 30 22:31:59.591671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.591568 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\" (UniqueName: \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\") pod \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\" (UID: \"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d\") " Oct 30 22:31:59.591866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.591843 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca" pod "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" (UID: "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:59.592158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.592140 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0bfc7d-7252-4559-a66e-38d50da85f68-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.592223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.592162 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.592223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.592177 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4xh7\" (UniqueName: \"kubernetes.io/projected/7a0bfc7d-7252-4559-a66e-38d50da85f68-kube-api-access-h4xh7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.593549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.593523 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca" pod "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" (UID: "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:59.593654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.593587 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh" pod "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" (UID: "c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d"). InnerVolumeSpecName "kube-api-access-jzlnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:59.693642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.693614 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.693642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:31:59.693642 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlnh\" (UniqueName: \"kubernetes.io/projected/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d-kube-api-access-jzlnh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:00.148544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.148508 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" path="/var/lib/kubelet/pods/690fa1e2-df19-4660-a8ae-f476f18a61f7/volumes" Oct 30 22:32:00.319915 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.319887 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" Oct 30 22:32:00.320305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.319915 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6" event={"ID":"c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d","Type":"ContainerDied","Data":"decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e"} Oct 30 22:32:00.320305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.319965 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="decd6c04f73bcdc25dce8d40af69de2343698c45865882f5e190eebd28a5d19e" Oct 30 22:32:00.321410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.321390 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" Oct 30 22:32:00.321528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.321429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl" event={"ID":"7a0bfc7d-7252-4559-a66e-38d50da85f68","Type":"ContainerDied","Data":"2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16"} Oct 30 22:32:00.321528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:00.321476 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2798b720045c89e9667a985f35ec5917c4ad1acc2856ec368a91927c858ccd16" Oct 30 22:32:02.256863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:02.256826 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:32:03.263793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:03.263765 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:32:09.220099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:09.220055 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:13.853941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:13.853902 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2"] Oct 30 22:32:13.904238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:13.904212 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7fd7ba-zx7p2"] Oct 30 22:32:14.147522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:14.147442 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88a5673-9173-4007-a966-90f6e20df6ec" path="/var/lib/kubelet/pods/c88a5673-9173-4007-a966-90f6e20df6ec/volumes" Oct 30 22:32:16.442313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.442275 4292 generic.go:356] "Generic (PLEG): container finished" podID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerID="7488f2acc2261f0533b9059774232f857448a9c827cd41b32e435f3ecb3450d2" exitCode=0 Oct 30 22:32:16.443147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.442325 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" event={"ID":"436426e1-cc3f-4212-9b7c-21bba88d61e1","Type":"ContainerDied","Data":"7488f2acc2261f0533b9059774232f857448a9c827cd41b32e435f3ecb3450d2"} Oct 30 22:32:16.497111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.497056 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:32:16.589921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.589894 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\") pod \"436426e1-cc3f-4212-9b7c-21bba88d61e1\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " Oct 30 22:32:16.590056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.590038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\" (UniqueName: \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\") pod \"436426e1-cc3f-4212-9b7c-21bba88d61e1\" (UID: \"436426e1-cc3f-4212-9b7c-21bba88d61e1\") " Oct 30 22:32:16.590299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.590274 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume" pod "436426e1-cc3f-4212-9b7c-21bba88d61e1" (UID: "436426e1-cc3f-4212-9b7c-21bba88d61e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:16.592062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.592037 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd" pod "436426e1-cc3f-4212-9b7c-21bba88d61e1" (UID: "436426e1-cc3f-4212-9b7c-21bba88d61e1"). InnerVolumeSpecName "kube-api-access-m9hwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:16.691094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.691064 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9hwd\" (UniqueName: \"kubernetes.io/projected/436426e1-cc3f-4212-9b7c-21bba88d61e1-kube-api-access-m9hwd\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.691094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:16.691091 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436426e1-cc3f-4212-9b7c-21bba88d61e1-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.451953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:17.451925 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" Oct 30 22:32:17.452386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:17.451917 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86" event={"ID":"436426e1-cc3f-4212-9b7c-21bba88d61e1","Type":"ContainerDied","Data":"61383ac686c2d300c5dc506468811f8b294a363880184b39770120f0c7f31ab3"} Oct 30 22:32:17.452386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:17.452064 4292 scope.go:122] "RemoveContainer" containerID="7488f2acc2261f0533b9059774232f857448a9c827cd41b32e435f3ecb3450d2" Oct 30 22:32:17.622487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:17.622457 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:32:17.672462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:17.672432 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-77cb1f-f8477558f-jpn86"] Oct 30 22:32:18.153869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:18.153838 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" path="/var/lib/kubelet/pods/436426e1-cc3f-4212-9b7c-21bba88d61e1/volumes" Oct 30 22:32:23.195320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:23.195288 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:32:23.195797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:23.195502 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" containerID="cri-o://fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316" gracePeriod=30 Oct 30 22:32:25.625215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:25.625179 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg"] Oct 30 22:32:25.675399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:25.675368 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b7588a-r7xhg"] Oct 30 22:32:26.148917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:26.148881 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d2e883-e7a0-487f-a078-eb55e661e7e7" path="/var/lib/kubelet/pods/10d2e883-e7a0-487f-a078-eb55e661e7e7/volumes" Oct 30 22:32:26.640526 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:32:26.640489 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:32:28.257895 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.257865 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k"] Oct 30 22:32:28.258443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258426 4292 cpu_manager.go:414] "containerMap: removing container" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" Oct 30 22:32:28.258487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258445 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" Oct 30 22:32:28.258487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258469 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" containerName="revoke-keys" Oct 30 22:32:28.258487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258475 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" containerName="revoke-keys" Oct 30 22:32:28.258487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258484 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7a0bfc7d-7252-4559-a66e-38d50da85f68" containerName="revoke-keys" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258489 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0bfc7d-7252-4559-a66e-38d50da85f68" containerName="revoke-keys" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258512 4292 cpu_manager.go:414] "containerMap: removing container" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258517 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258623 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="690fa1e2-df19-4660-a8ae-f476f18a61f7" containerName="main" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258635 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="436426e1-cc3f-4212-9b7c-21bba88d61e1" containerName="payload-processing" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258644 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" containerName="revoke-keys" Oct 30 22:32:28.258661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.258652 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0bfc7d-7252-4559-a66e-38d50da85f68" containerName="revoke-keys" Oct 30 22:32:28.262155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.262139 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.320593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.320551 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:28.338797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.338764 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.338962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.338878 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvzg\" (UniqueName: \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.338962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.338931 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.368202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.368165 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:32:28.371170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.371142 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:28.371313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.371191 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:28.372441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.372420 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k"] Oct 30 22:32:28.440723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.440682 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.440911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.440749 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvzg\" (UniqueName: \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.440911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.440893 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.441555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.441532 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.443776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.443753 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.470956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.470904 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:32:28.509770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.509746 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvzg\" (UniqueName: \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\") pod \"maas-api-revoke-sub-1ffc4eb2-qnw4k\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.539639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.539615 4292 generic.go:356] "Generic (PLEG): container finished" podID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerID="fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316" exitCode=0 Oct 30 22:32:28.539732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.539664 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" Oct 30 22:32:28.539732 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.539690 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerDied","Data":"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316"} Oct 30 22:32:28.539806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.539746 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp" event={"ID":"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72","Type":"ContainerDied","Data":"00735abe1760ef5329bd2bfc842089d18eadea428d24bd03b6029c25a228457a"} Oct 30 22:32:28.539806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.539769 4292 scope.go:122] "RemoveContainer" containerID="fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316" Oct 30 22:32:28.542033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542010 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542091 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542131 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542219 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542294 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542269 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542295 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542327 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542396 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542365 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:28.542604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542471 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:28.542604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542500 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:28.542604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542519 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\") pod \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\" (UID: \"3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72\") " Oct 30 22:32:28.542785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542718 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:28.542785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.542757 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:28.543093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.543066 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.543198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.543091 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.543198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.543121 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.543198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.543137 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.543198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.543151 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.544467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.544442 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:28.544596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.544482 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:28.544873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.544851 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:28.544969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.544953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj" pod "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" (UID: "3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72"). InnerVolumeSpecName "kube-api-access-4w8gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:28.550266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.550243 4292 scope.go:122] "RemoveContainer" containerID="fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5" Oct 30 22:32:28.560973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.560950 4292 scope.go:122] "RemoveContainer" containerID="fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316" Oct 30 22:32:28.561270 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:32:28.561241 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316\": container with ID starting with fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316 not found: ID does not exist" containerID="fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316" Oct 30 22:32:28.561318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.561278 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316"} err="failed to get container status \"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316\": rpc error: code = NotFound desc = could not find container \"fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316\": container with ID starting with fc1f9c136ed2e1fc34f601c7427ba7caaff3cbe8cc16875c3956ce50eff5f316 not found: ID does not exist" Oct 30 22:32:28.561318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.561294 4292 scope.go:122] "RemoveContainer" containerID="fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5" Oct 30 22:32:28.561555 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:32:28.561537 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5\": container with ID starting with fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5 not found: ID does not exist" containerID="fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5" Oct 30 22:32:28.561624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.561562 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5"} err="failed to get container status \"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5\": rpc error: code = NotFound desc = could not find container \"fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5\": container with ID starting with fa8fbf26aeebf0cf87b17517391d0824d22bbe251b45c9b2bb7b88a4e8be23d5 not found: ID does not exist" Oct 30 22:32:28.580513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.580493 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:28.647759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.644497 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8gj\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-kube-api-access-4w8gj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.647759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.644525 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.647759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.644538 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.647759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.644548 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.962674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.962646 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k"] Oct 30 22:32:28.963292 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:32:28.963269 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e3ea87_67c5_451e_97ef_52612bb25713.slice/crio-f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac WatchSource:0}: Error finding container f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac: Status 404 returned error can't find the container with id f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac Oct 30 22:32:28.968054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.968034 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:32:28.973286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:28.973265 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7fd7ba-openshift-default-76bd897d7-gkkmp"] Oct 30 22:32:29.549062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:29.549023 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" event={"ID":"02e3ea87-67c5-451e-97ef-52612bb25713","Type":"ContainerStarted","Data":"bf07db9cfc8d7bca52f5314c5cd98e3e1490bf7f81e3312d8e1103bde1c008c2"} Oct 30 22:32:29.549533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:29.549081 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" event={"ID":"02e3ea87-67c5-451e-97ef-52612bb25713","Type":"ContainerStarted","Data":"f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac"} Oct 30 22:32:29.676615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:29.671611 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" podStartSLOduration=1.671559388 podStartE2EDuration="1.671559388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:29.669851215 +0000 UTC m=+2277.638670779" watchObservedRunningTime="2026-10-30 22:32:29.671559388 +0000 UTC m=+2277.640378953" Oct 30 22:32:30.148348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.148313 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" path="/var/lib/kubelet/pods/3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72/volumes" Oct 30 22:32:30.220738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.220669 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j"] Oct 30 22:32:30.221230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221217 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.221278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221232 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.221278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221245 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.221278 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221250 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.221370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221336 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.221370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.221343 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b14f4ec-bdc4-48c8-b06e-dd1a3f3efc72" containerName="istio-proxy" Oct 30 22:32:30.224763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.224747 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.265605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.265563 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt"] Oct 30 22:32:30.269456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.269434 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.284446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.284420 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j"] Oct 30 22:32:30.332897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.332865 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt"] Oct 30 22:32:30.371059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371025 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.371059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371060 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.371327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371080 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.371327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371259 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.371327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371309 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.371512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.371444 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvj2\" (UniqueName: \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.473018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.472939 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.473018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.472976 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.473018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.472996 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.473298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.473042 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.473298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.473065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.473298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.473180 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvj2\" (UniqueName: \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.473838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.473810 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.473956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.473837 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.475380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.475360 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.475492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.475476 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.536951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.536921 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvj2\" (UniqueName: \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\") pod \"maas-api-revoke-sub-50aa7e7a-wddzt\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.536951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.536935 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\") pod \"maas-api-revoke-sub-35778301-tfr5j\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.537554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.537536 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:30.557705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.557677 4292 generic.go:356] "Generic (PLEG): container finished" podID="02e3ea87-67c5-451e-97ef-52612bb25713" containerID="bf07db9cfc8d7bca52f5314c5cd98e3e1490bf7f81e3312d8e1103bde1c008c2" exitCode=0 Oct 30 22:32:30.558010 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.557724 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" event={"ID":"02e3ea87-67c5-451e-97ef-52612bb25713","Type":"ContainerDied","Data":"bf07db9cfc8d7bca52f5314c5cd98e3e1490bf7f81e3312d8e1103bde1c008c2"} Oct 30 22:32:30.583688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.583660 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:30.951492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.951447 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j"] Oct 30 22:32:30.954154 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:32:30.954127 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f592536_365a_4c91_9a4e_913a445e9e4f.slice/crio-4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706 WatchSource:0}: Error finding container 4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706: Status 404 returned error can't find the container with id 4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706 Oct 30 22:32:30.988553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:30.988521 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt"] Oct 30 22:32:30.989207 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:32:30.989182 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1187f480_c592_4a8c_964f_b3dab2215144.slice/crio-1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29 WatchSource:0}: Error finding container 1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29: Status 404 returned error can't find the container with id 1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29 Oct 30 22:32:31.566714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.566675 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" event={"ID":"2f592536-365a-4c91-9a4e-913a445e9e4f","Type":"ContainerStarted","Data":"59e70da7ac7ea891c62b4c5a4ff0cc9eeebd303ed5920fd7813e18e3ef2e5594"} Oct 30 22:32:31.567179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.566744 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" event={"ID":"2f592536-365a-4c91-9a4e-913a445e9e4f","Type":"ContainerStarted","Data":"4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706"} Oct 30 22:32:31.568603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.568545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" event={"ID":"1187f480-c592-4a8c-964f-b3dab2215144","Type":"ContainerStarted","Data":"880317f7835224fd1706c6246a267ca49424ceebfc7e1c6479bfecbd30b8ca33"} Oct 30 22:32:31.568722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.568613 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" event={"ID":"1187f480-c592-4a8c-964f-b3dab2215144","Type":"ContainerStarted","Data":"1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29"} Oct 30 22:32:31.688664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.688609 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" podStartSLOduration=1.688595023 podStartE2EDuration="1.688595023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:31.686492698 +0000 UTC m=+2279.655312261" watchObservedRunningTime="2026-10-30 22:32:31.688595023 +0000 UTC m=+2279.657414587" Oct 30 22:32:31.731059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.731034 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:31.806837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.806780 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" podStartSLOduration=1.806762081 podStartE2EDuration="1.806762081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:31.805159167 +0000 UTC m=+2279.773978732" watchObservedRunningTime="2026-10-30 22:32:31.806762081 +0000 UTC m=+2279.775581647" Oct 30 22:32:31.908188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.908162 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\") pod \"02e3ea87-67c5-451e-97ef-52612bb25713\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " Oct 30 22:32:31.908320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.908304 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\" (UniqueName: \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\") pod \"02e3ea87-67c5-451e-97ef-52612bb25713\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " Oct 30 22:32:31.908392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.908374 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\") pod \"02e3ea87-67c5-451e-97ef-52612bb25713\" (UID: \"02e3ea87-67c5-451e-97ef-52612bb25713\") " Oct 30 22:32:31.908557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.908534 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca" pod "02e3ea87-67c5-451e-97ef-52612bb25713" (UID: "02e3ea87-67c5-451e-97ef-52612bb25713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:31.908857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.908837 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.910211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.910181 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca" pod "02e3ea87-67c5-451e-97ef-52612bb25713" (UID: "02e3ea87-67c5-451e-97ef-52612bb25713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:31.910303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:31.910279 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg" pod "02e3ea87-67c5-451e-97ef-52612bb25713" (UID: "02e3ea87-67c5-451e-97ef-52612bb25713"). InnerVolumeSpecName "kube-api-access-xpvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:32.010249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.010217 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvzg\" (UniqueName: \"kubernetes.io/projected/02e3ea87-67c5-451e-97ef-52612bb25713-kube-api-access-xpvzg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:32.010249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.010244 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3ea87-67c5-451e-97ef-52612bb25713-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:32.577782 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.577744 4292 generic.go:356] "Generic (PLEG): container finished" podID="1187f480-c592-4a8c-964f-b3dab2215144" containerID="880317f7835224fd1706c6246a267ca49424ceebfc7e1c6479bfecbd30b8ca33" exitCode=0 Oct 30 22:32:32.578159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.577781 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" event={"ID":"1187f480-c592-4a8c-964f-b3dab2215144","Type":"ContainerDied","Data":"880317f7835224fd1706c6246a267ca49424ceebfc7e1c6479bfecbd30b8ca33"} Oct 30 22:32:32.579444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.579420 4292 generic.go:356] "Generic (PLEG): container finished" podID="2f592536-365a-4c91-9a4e-913a445e9e4f" containerID="59e70da7ac7ea891c62b4c5a4ff0cc9eeebd303ed5920fd7813e18e3ef2e5594" exitCode=0 Oct 30 22:32:32.579591 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.579511 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" event={"ID":"2f592536-365a-4c91-9a4e-913a445e9e4f","Type":"ContainerDied","Data":"59e70da7ac7ea891c62b4c5a4ff0cc9eeebd303ed5920fd7813e18e3ef2e5594"} Oct 30 22:32:32.580882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.580866 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" Oct 30 22:32:32.580970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.580884 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k" event={"ID":"02e3ea87-67c5-451e-97ef-52612bb25713","Type":"ContainerDied","Data":"f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac"} Oct 30 22:32:32.580970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:32.580925 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90c8a3f453f571f02d1ec17027b3073cd7f2e2a3047a2566542cb7f97c7f0ac" Oct 30 22:32:33.759406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.759377 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:33.789567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.789543 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:33.937426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937340 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\") pod \"2f592536-365a-4c91-9a4e-913a445e9e4f\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " Oct 30 22:32:33.937426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937402 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\") pod \"1187f480-c592-4a8c-964f-b3dab2215144\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " Oct 30 22:32:33.937662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937433 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\" (UniqueName: \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\") pod \"1187f480-c592-4a8c-964f-b3dab2215144\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " Oct 30 22:32:33.937662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937462 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\") pod \"2f592536-365a-4c91-9a4e-913a445e9e4f\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " Oct 30 22:32:33.937884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937856 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca" pod "2f592536-365a-4c91-9a4e-913a445e9e4f" (UID: "2f592536-365a-4c91-9a4e-913a445e9e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:33.937884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937864 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca" pod "1187f480-c592-4a8c-964f-b3dab2215144" (UID: "1187f480-c592-4a8c-964f-b3dab2215144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:33.938080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937505 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\") pod \"2f592536-365a-4c91-9a4e-913a445e9e4f\" (UID: \"2f592536-365a-4c91-9a4e-913a445e9e4f\") " Oct 30 22:32:33.938080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.937936 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\") pod \"1187f480-c592-4a8c-964f-b3dab2215144\" (UID: \"1187f480-c592-4a8c-964f-b3dab2215144\") " Oct 30 22:32:33.938465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.938441 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:33.938595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.938467 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:33.939791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.939763 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca" pod "2f592536-365a-4c91-9a4e-913a445e9e4f" (UID: "2f592536-365a-4c91-9a4e-913a445e9e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:33.940057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.940029 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca" pod "1187f480-c592-4a8c-964f-b3dab2215144" (UID: "1187f480-c592-4a8c-964f-b3dab2215144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:33.940140 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.940087 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls" pod "2f592536-365a-4c91-9a4e-913a445e9e4f" (UID: "2f592536-365a-4c91-9a4e-913a445e9e4f"). InnerVolumeSpecName "kube-api-access-nmzls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:33.941093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:33.941072 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2" pod "1187f480-c592-4a8c-964f-b3dab2215144" (UID: "1187f480-c592-4a8c-964f-b3dab2215144"). InnerVolumeSpecName "kube-api-access-4mvj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:34.039641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.039609 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f592536-365a-4c91-9a4e-913a445e9e4f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:34.039641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.039637 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvj2\" (UniqueName: \"kubernetes.io/projected/1187f480-c592-4a8c-964f-b3dab2215144-kube-api-access-4mvj2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:34.039858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.039650 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/2f592536-365a-4c91-9a4e-913a445e9e4f-kube-api-access-nmzls\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:34.039858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.039659 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1187f480-c592-4a8c-964f-b3dab2215144-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:34.606691 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.606660 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" Oct 30 22:32:34.606881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.606691 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt" event={"ID":"1187f480-c592-4a8c-964f-b3dab2215144","Type":"ContainerDied","Data":"1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29"} Oct 30 22:32:34.606881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.606737 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c96b6e377198582414ea9b1f3dc22a41765d49c447c53b960e07725deb85b29" Oct 30 22:32:34.608139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.608118 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" Oct 30 22:32:34.608139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.608128 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j" event={"ID":"2f592536-365a-4c91-9a4e-913a445e9e4f","Type":"ContainerDied","Data":"4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706"} Oct 30 22:32:34.608344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:34.608167 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4666ed678b16ea3110ce67da2c038c3015db7cca97da316cbf5fec1400438706" Oct 30 22:32:37.316980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:37.316944 4292 scope.go:122] "RemoveContainer" containerID="659d9c7f2f7a3890ecaa3500a97df09d3ebe0a70271e68fc056442db496c27c4" Oct 30 22:32:53.513041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:53.512961 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8"] Oct 30 22:32:53.563360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:53.563329 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-13b8d8-746z8"] Oct 30 22:32:54.146365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:32:54.146328 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b619ea-60fd-41de-9b26-bbd2c98de9a3" path="/var/lib/kubelet/pods/64b619ea-60fd-41de-9b26-bbd2c98de9a3/volumes" Oct 30 22:32:56.639943 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:32:56.639898 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:33:01.910361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:01.910328 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:33:01.910938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:01.910605 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" containerID="cri-o://b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80" gracePeriod=30 Oct 30 22:33:07.216647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.216564 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:33:07.229436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229409 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229612 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229500 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229690 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:07.229752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229735 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229769 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229824 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.229877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229841 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:07.229877 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229848 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.230112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229910 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.230112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.229946 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\") pod \"7fc5c76c-3473-4619-a66a-096f1e769f67\" (UID: \"7fc5c76c-3473-4619-a66a-096f1e769f67\") " Oct 30 22:33:07.230222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.230100 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:07.230280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.230224 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:07.230738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.230474 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:07.231190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.231004 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.231190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.231132 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.231190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.231153 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc5c76c-3473-4619-a66a-096f1e769f67-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.231190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.231165 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.231190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.231177 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.232168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.232146 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:07.232393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.232375 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "kube-api-access-bswmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:07.232475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.232461 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:07.232935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.232911 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy" pod "7fc5c76c-3473-4619-a66a-096f1e769f67" (UID: "7fc5c76c-3473-4619-a66a-096f1e769f67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:07.332052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.332022 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.332052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.332048 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.332052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.332057 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswmw\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-kube-api-access-bswmw\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.332270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.332066 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc5c76c-3473-4619-a66a-096f1e769f67-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:07.920422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.920384 4292 generic.go:356] "Generic (PLEG): container finished" podID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerID="b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80" exitCode=0 Oct 30 22:33:07.920616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.920437 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" Oct 30 22:33:07.920616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.920476 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" event={"ID":"7fc5c76c-3473-4619-a66a-096f1e769f67","Type":"ContainerDied","Data":"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80"} Oct 30 22:33:07.920616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.920536 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn" event={"ID":"7fc5c76c-3473-4619-a66a-096f1e769f67","Type":"ContainerDied","Data":"7d30f3319c31d75178d7003c560e7151575b73764f020751b5322c925f24328e"} Oct 30 22:33:07.920616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.920562 4292 scope.go:122] "RemoveContainer" containerID="b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80" Oct 30 22:33:07.931334 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.931317 4292 scope.go:122] "RemoveContainer" containerID="b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80" Oct 30 22:33:07.931613 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:07.931590 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80\": container with ID starting with b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80 not found: ID does not exist" containerID="b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80" Oct 30 22:33:07.931669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:07.931620 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80"} err="failed to get container status \"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80\": rpc error: code = NotFound desc = could not find container \"b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80\": container with ID starting with b040b94dafd9e4292e596c936e3dd4b8bf42276b923d1d10bb9758f891aa3e80 not found: ID does not exist" Oct 30 22:33:08.111072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:08.111029 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:33:08.116854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:08.116821 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-13b8d8-openshift-default-55bd664645-mq9kn"] Oct 30 22:33:08.147046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:08.147011 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" path="/var/lib/kubelet/pods/7fc5c76c-3473-4619-a66a-096f1e769f67/volumes" Oct 30 22:33:13.218748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.218708 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:13.225327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.224949 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:13.479452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.479369 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:33:13.479992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.479975 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2f592536-365a-4c91-9a4e-913a445e9e4f" containerName="revoke-keys" Oct 30 22:33:13.480046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.479994 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f592536-365a-4c91-9a4e-913a445e9e4f" containerName="revoke-keys" Oct 30 22:33:13.480046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480015 4292 cpu_manager.go:414] "containerMap: removing container" podUID="02e3ea87-67c5-451e-97ef-52612bb25713" containerName="revoke-keys" Oct 30 22:33:13.480046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480019 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e3ea87-67c5-451e-97ef-52612bb25713" containerName="revoke-keys" Oct 30 22:33:13.480046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480030 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" Oct 30 22:33:13.480046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480034 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1187f480-c592-4a8c-964f-b3dab2215144" containerName="revoke-keys" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480057 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1187f480-c592-4a8c-964f-b3dab2215144" containerName="revoke-keys" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480142 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e3ea87-67c5-451e-97ef-52612bb25713" containerName="revoke-keys" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480153 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc5c76c-3473-4619-a66a-096f1e769f67" containerName="istio-proxy" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480162 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1187f480-c592-4a8c-964f-b3dab2215144" containerName="revoke-keys" Oct 30 22:33:13.480210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.480172 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f592536-365a-4c91-9a4e-913a445e9e4f" containerName="revoke-keys" Oct 30 22:33:13.483329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.483313 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.608394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.608359 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609483 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609540 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609569 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609717 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609760 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609806 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609835 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609888 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsfc\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.610177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.609945 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711257 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711309 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711339 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711453 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711496 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711544 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711592 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhsfc\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.711870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.711793 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.712054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.712029 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.712139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.712109 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.712387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.712364 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.712429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.712365 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.713676 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.713656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.713914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.713895 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.777231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.777176 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.777324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.777292 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhsfc\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\") pod \"maas-default-gateway-openshift-default-76597d6855-5k6lk\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.796856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.796832 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:13.964721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.964688 4292 generic.go:356] "Generic (PLEG): container finished" podID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" exitCode=0 Oct 30 22:33:13.964887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.964757 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerDied","Data":"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d"} Oct 30 22:33:13.965213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.965194 4292 scope.go:122] "RemoveContainer" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" Oct 30 22:33:13.966773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.966747 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:13.966858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.966817 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:13.966858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:13.966850 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:14.267410 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:33:14.267362 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4111_604b_4ff3_86a1_81d42059a42d.slice/crio-f352017f077961d2f8ce1b8b07057ade2fa4c664c46ac5157fdb9c2c11f5f004 WatchSource:0}: Error finding container f352017f077961d2f8ce1b8b07057ade2fa4c664c46ac5157fdb9c2c11f5f004: Status 404 returned error can't find the container with id f352017f077961d2f8ce1b8b07057ade2fa4c664c46ac5157fdb9c2c11f5f004 Oct 30 22:33:14.268563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.268534 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:33:14.270422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.270391 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:14.270530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.270452 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:14.270530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.270478 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:14.975000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.974948 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerStarted","Data":"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b"} Oct 30 22:33:14.976814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.976785 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" event={"ID":"13eb4111-604b-4ff3-86a1-81d42059a42d","Type":"ContainerStarted","Data":"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f"} Oct 30 22:33:14.976977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:14.976836 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" event={"ID":"13eb4111-604b-4ff3-86a1-81d42059a42d","Type":"ContainerStarted","Data":"f352017f077961d2f8ce1b8b07057ade2fa4c664c46ac5157fdb9c2c11f5f004"} Oct 30 22:33:15.241289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.241182 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podStartSLOduration=2.241165669 podStartE2EDuration="2.241165669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:15.24021536 +0000 UTC m=+2323.209034924" watchObservedRunningTime="2026-10-30 22:33:15.241165669 +0000 UTC m=+2323.209985233" Oct 30 22:33:15.799379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.799344 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:15.800090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.800065 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:15.800283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.800254 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:15.941174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.941115 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:33:15.941689 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.941665 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:15.941799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:15.941711 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:16.807599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:16.805179 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:16.807599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:16.805236 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:16.941826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:16.941790 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:16.942015 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:16.941845 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:17.800601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:17.797549 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:17.800601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:17.797641 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:17.941703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:17.941662 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:17.942070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:17.941742 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:18.800967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:18.800929 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:18.801260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:18.801231 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:18.943595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:18.943037 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:18.943595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:18.943114 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:19.801935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:19.801895 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:19.802227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:19.802200 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:19.943178 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:19.943141 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:33:19.943484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:19.943452 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 22:33:20.500203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.500159 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-2f9382"] Oct 30 22:33:20.504494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.504472 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:20.562408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.562380 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-2f9382"] Oct 30 22:33:20.607256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.607224 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tmcpb\"" Oct 30 22:33:20.636038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.636014 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tnp\" (UniqueName: \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\") pod \"test-curl-7438-2f9382\" (UID: \"aaa13014-9c93-4be5-8fb2-2c284e488c72\") " pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:20.737397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.737359 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tnp\" (UniqueName: \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\") pod \"test-curl-7438-2f9382\" (UID: \"aaa13014-9c93-4be5-8fb2-2c284e488c72\") " pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:20.797222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.797193 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:20.797354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.797249 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:20.800853 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.800827 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tnp\" (UniqueName: \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\") pod \"test-curl-7438-2f9382\" (UID: \"aaa13014-9c93-4be5-8fb2-2c284e488c72\") " pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:20.817234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.817215 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:20.945651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.945622 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:33:20.946125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.946106 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:33:20.946876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:20.946854 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:33:21.199154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.199126 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-2f9382"] Oct 30 22:33:21.199975 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:33:21.199953 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaa13014_9c93_4be5_8fb2_2c284e488c72.slice/crio-de7fcfe1707b6698051d684e413259ecff5c5abc72b145e919e7cdf45ecdd8a7 WatchSource:0}: Error finding container de7fcfe1707b6698051d684e413259ecff5c5abc72b145e919e7cdf45ecdd8a7: Status 404 returned error can't find the container with id de7fcfe1707b6698051d684e413259ecff5c5abc72b145e919e7cdf45ecdd8a7 Oct 30 22:33:21.203888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.203866 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:33:21.366815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.366779 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq"] Oct 30 22:33:21.373418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.373389 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-77cb1f-s4gcq"] Oct 30 22:33:21.797759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.797721 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 22:33:21.798165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:21.797786 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 22:33:22.081314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:22.081186 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-2f9382" event={"ID":"aaa13014-9c93-4be5-8fb2-2c284e488c72","Type":"ContainerStarted","Data":"de7fcfe1707b6698051d684e413259ecff5c5abc72b145e919e7cdf45ecdd8a7"} Oct 30 22:33:22.148825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:22.148787 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011d6f96-99bf-4a35-aa53-75d438f0a58e" path="/var/lib/kubelet/pods/011d6f96-99bf-4a35-aa53-75d438f0a58e/volumes" Oct 30 22:33:22.953985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:22.953948 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:22.954694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:22.954655 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:22.956182 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:22.956155 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:33:25.110543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:25.110507 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-2f9382" event={"ID":"aaa13014-9c93-4be5-8fb2-2c284e488c72","Type":"ContainerStarted","Data":"9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66"} Oct 30 22:33:25.229570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:25.229520 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7438-2f9382" podStartSLOduration=2.239363961 podStartE2EDuration="5.229506032s" totalImagesPullingTime="2.990142071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:25.226956356 +0000 UTC m=+2333.195775897" watchObservedRunningTime="2026-10-30 22:33:25.229506032 +0000 UTC m=+2333.198325595" Oct 30 22:33:26.642132 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:26.642090 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:33:26.840464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:26.840429 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-2f9382"] Oct 30 22:33:26.847262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:26.847229 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-2f9382"] Oct 30 22:33:27.122858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.122818 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7438-2f9382" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" containerName="test-curl-7438-2f9382" containerID="cri-o://9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66" gracePeriod=2 Oct 30 22:33:27.174724 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:27.174685 4292 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7438-2f9382\" is forbidden: User \"system:node:ip-10-0-142-210.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-210.us-west-1.compute.internal' and this object" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:27.424912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.424841 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj"] Oct 30 22:33:27.425430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.425417 4292 cpu_manager.go:414] "containerMap: removing container" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" containerName="test-curl-7438-2f9382" Oct 30 22:33:27.425473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.425431 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" containerName="test-curl-7438-2f9382" Oct 30 22:33:27.425545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.425535 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" containerName="test-curl-7438-2f9382" Oct 30 22:33:27.429026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.429011 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.484485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.484457 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:27.484894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.484852 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:27.488109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.488080 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj"] Oct 30 22:33:27.524606 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.524560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.524718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.524652 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj979\" (UniqueName: \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.524784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.524773 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.534251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.534232 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:33:27.534363 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.534271 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:27.598659 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:27.598631 4292 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7438-2f9382\" is forbidden: User \"system:node:ip-10-0-142-210.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-210.us-west-1.compute.internal' and this object" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:27.625652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.625629 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.625764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.625671 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj979\" (UniqueName: \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.625829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.625812 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.626313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.626294 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.628188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.628170 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.689085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.689032 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj979\" (UniqueName: \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\") pod \"maas-api-revoke-keys-e2e-mt-18515933-9nscj\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:27.742844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:27.742823 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:28.123286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:28.123255 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj"] Oct 30 22:33:28.124446 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:33:28.124415 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dc64bf_8a9b_405b_9826_bb5b155a03a4.slice/crio-4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326 WatchSource:0}: Error finding container 4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326: Status 404 returned error can't find the container with id 4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326 Oct 30 22:33:29.138118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.138083 4292 generic.go:356] "Generic (PLEG): container finished" podID="a5dc64bf-8a9b-405b-9826-bb5b155a03a4" containerID="97a1ca69647679aaab70ae165ee2c33e71e215800c29b70c786866670d773134" exitCode=0 Oct 30 22:33:29.138524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.138172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" event={"ID":"a5dc64bf-8a9b-405b-9826-bb5b155a03a4","Type":"ContainerDied","Data":"97a1ca69647679aaab70ae165ee2c33e71e215800c29b70c786866670d773134"} Oct 30 22:33:29.138524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.138244 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" event={"ID":"a5dc64bf-8a9b-405b-9826-bb5b155a03a4","Type":"ContainerStarted","Data":"4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326"} Oct 30 22:33:29.379359 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.379334 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:29.450217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.450182 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\" (UniqueName: \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\") pod \"aaa13014-9c93-4be5-8fb2-2c284e488c72\" (UID: \"aaa13014-9c93-4be5-8fb2-2c284e488c72\") " Oct 30 22:33:29.452283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.452253 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp" pod "aaa13014-9c93-4be5-8fb2-2c284e488c72" (UID: "aaa13014-9c93-4be5-8fb2-2c284e488c72"). InnerVolumeSpecName "kube-api-access-n2tnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:29.552217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:29.552177 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2tnp\" (UniqueName: \"kubernetes.io/projected/aaa13014-9c93-4be5-8fb2-2c284e488c72-kube-api-access-n2tnp\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:30.011082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.010989 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:33:30.011312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.011280 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" containerID="cri-o://ca4355df6310863ded04367f99790fa24a6a5ea4dfb906baaf5c203ca4726fb0" gracePeriod=30 Oct 30 22:33:30.035114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.035085 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:33:30.035303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.035282 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" containerID="cri-o://e9e8143430e66ad61c90e63ca17887d0ce1087691cb8c14f2cb6ee10af72c812" gracePeriod=30 Oct 30 22:33:30.172000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.169974 4292 generic.go:356] "Generic (PLEG): container finished" podID="aaa13014-9c93-4be5-8fb2-2c284e488c72" containerID="9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66" exitCode=137 Oct 30 22:33:30.172000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.170251 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-2f9382" Oct 30 22:33:30.173266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.173239 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa13014-9c93-4be5-8fb2-2c284e488c72" path="/var/lib/kubelet/pods/aaa13014-9c93-4be5-8fb2-2c284e488c72/volumes" Oct 30 22:33:30.173781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.173764 4292 scope.go:122] "RemoveContainer" containerID="9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66" Oct 30 22:33:30.191222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.191201 4292 scope.go:122] "RemoveContainer" containerID="9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66" Oct 30 22:33:30.191537 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:30.191511 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66\": container with ID starting with 9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66 not found: ID does not exist" containerID="9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66" Oct 30 22:33:30.191662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.191549 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66"} err="failed to get container status \"9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66\": rpc error: code = NotFound desc = could not find container \"9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66\": container with ID starting with 9ad0da05374b08c88a09534e398f9bfc1cd45d7f06eb9d041b7382b918050a66 not found: ID does not exist" Oct 30 22:33:30.333789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.333767 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:30.464984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.464948 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\" (UniqueName: \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\") pod \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " Oct 30 22:33:30.465166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.465100 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\") pod \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " Oct 30 22:33:30.465241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.465165 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\") pod \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\" (UID: \"a5dc64bf-8a9b-405b-9826-bb5b155a03a4\") " Oct 30 22:33:30.465568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.465543 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca" pod "a5dc64bf-8a9b-405b-9826-bb5b155a03a4" (UID: "a5dc64bf-8a9b-405b-9826-bb5b155a03a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:30.465730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.465707 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:30.467170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.467143 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979" pod "a5dc64bf-8a9b-405b-9826-bb5b155a03a4" (UID: "a5dc64bf-8a9b-405b-9826-bb5b155a03a4"). InnerVolumeSpecName "kube-api-access-vj979". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:30.467651 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.467623 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca" pod "a5dc64bf-8a9b-405b-9826-bb5b155a03a4" (UID: "a5dc64bf-8a9b-405b-9826-bb5b155a03a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:30.566811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.566736 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj979\" (UniqueName: \"kubernetes.io/projected/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-kube-api-access-vj979\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:30.566811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:30.566772 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc64bf-8a9b-405b-9826-bb5b155a03a4-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.179719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:31.179691 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" Oct 30 22:33:31.180108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:31.179725 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj" event={"ID":"a5dc64bf-8a9b-405b-9826-bb5b155a03a4","Type":"ContainerDied","Data":"4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326"} Oct 30 22:33:31.180108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:31.179773 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3e72909b57a1ad3e85f9a28cd4879c38fbf03ff483f80bbfcbfe7252756326" Oct 30 22:33:35.210257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.210209 4292 generic.go:356] "Generic (PLEG): container finished" podID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerID="e9e8143430e66ad61c90e63ca17887d0ce1087691cb8c14f2cb6ee10af72c812" exitCode=0 Oct 30 22:33:35.210736 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.210366 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" event={"ID":"42c275a4-0c6f-469a-815d-d0cb228b5cc0","Type":"ContainerDied","Data":"e9e8143430e66ad61c90e63ca17887d0ce1087691cb8c14f2cb6ee10af72c812"} Oct 30 22:33:35.213053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.212919 4292 generic.go:356] "Generic (PLEG): container finished" podID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerID="ca4355df6310863ded04367f99790fa24a6a5ea4dfb906baaf5c203ca4726fb0" exitCode=0 Oct 30 22:33:35.213053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.212981 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" event={"ID":"23078e7c-36a3-4bb0-a582-b72f4faad4b0","Type":"ContainerDied","Data":"ca4355df6310863ded04367f99790fa24a6a5ea4dfb906baaf5c203ca4726fb0"} Oct 30 22:33:35.301025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.300996 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:33:35.327485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.327464 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:33:35.421464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421431 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421470 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421509 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.421600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421723 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.421776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421751 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.421944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421794 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421846 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.421944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421878 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.421944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421922 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.421958 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422022 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422046 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422054 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422072 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422105 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422143 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422219 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\") pod \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\" (UID: \"42c275a4-0c6f-469a-815d-d0cb228b5cc0\") " Oct 30 22:33:35.422633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\") pod \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\" (UID: \"23078e7c-36a3-4bb0-a582-b72f4faad4b0\") " Oct 30 22:33:35.422633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422143 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.422888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422710 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.422888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422841 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.422888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.422863 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.423073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423015 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423025 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.423073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423043 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423060 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423075 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423088 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423091 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:35.423309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423102 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.423309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.423118 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.424697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.424672 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:35.424697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.424690 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.424865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.424835 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "kube-api-access-s7fln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:35.424931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.424914 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:35.424973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.424939 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "kube-api-access-vf5r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:35.425131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.425112 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:35.425196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.425139 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy" pod "42c275a4-0c6f-469a-815d-d0cb228b5cc0" (UID: "42c275a4-0c6f-469a-815d-d0cb228b5cc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:35.426034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.426014 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo" pod "23078e7c-36a3-4bb0-a582-b72f4faad4b0" (UID: "23078e7c-36a3-4bb0-a582-b72f4faad4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:35.524293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524262 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524296 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524311 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524328 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5r8\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-kube-api-access-vf5r8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524371 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524384 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524396 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524412 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fln\" (UniqueName: \"kubernetes.io/projected/23078e7c-36a3-4bb0-a582-b72f4faad4b0-kube-api-access-s7fln\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524428 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c275a4-0c6f-469a-815d-d0cb228b5cc0-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524442 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23078e7c-36a3-4bb0-a582-b72f4faad4b0-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:35.524917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:35.524456 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23078e7c-36a3-4bb0-a582-b72f4faad4b0-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:36.221081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.221052 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" Oct 30 22:33:36.221461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.221086 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9" event={"ID":"23078e7c-36a3-4bb0-a582-b72f4faad4b0","Type":"ContainerDied","Data":"ec96693e0bab4b47952e4deb6c32e277572be68655238992f3e66272d61d9d4e"} Oct 30 22:33:36.221461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.221146 4292 scope.go:122] "RemoveContainer" containerID="ca4355df6310863ded04367f99790fa24a6a5ea4dfb906baaf5c203ca4726fb0" Oct 30 22:33:36.222719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.222699 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" event={"ID":"42c275a4-0c6f-469a-815d-d0cb228b5cc0","Type":"ContainerDied","Data":"e543ca179cb38e4f04426a58fdd89dfb78e5a7b411a1f6d2662e747613e7cb0c"} Oct 30 22:33:36.222821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.222723 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx" Oct 30 22:33:36.231617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.231599 4292 scope.go:122] "RemoveContainer" containerID="e9e8143430e66ad61c90e63ca17887d0ce1087691cb8c14f2cb6ee10af72c812" Oct 30 22:33:36.394997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.394929 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:33:36.447520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.447493 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-qk4cx"] Oct 30 22:33:36.518917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.518886 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:33:36.527305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:36.527283 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-77cb1f-openshift-default-57f8c95f4c-jj5w9"] Oct 30 22:33:37.371304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:37.371272 4292 scope.go:122] "RemoveContainer" containerID="f8917b8a39b615b7e01d94afa7ccb46e70e1dabf1c34a8052ab2005dccee82f2" Oct 30 22:33:38.147336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.147304 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" path="/var/lib/kubelet/pods/23078e7c-36a3-4bb0-a582-b72f4faad4b0/volumes" Oct 30 22:33:38.148640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.148616 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" path="/var/lib/kubelet/pods/42c275a4-0c6f-469a-815d-d0cb228b5cc0/volumes" Oct 30 22:33:38.314873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.314836 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:33:38.315456 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315436 4292 cpu_manager.go:414] "containerMap: removing container" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" Oct 30 22:33:38.315565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315457 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" Oct 30 22:33:38.315565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315490 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a5dc64bf-8a9b-405b-9826-bb5b155a03a4" containerName="revoke-keys" Oct 30 22:33:38.315565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315498 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5dc64bf-8a9b-405b-9826-bb5b155a03a4" containerName="revoke-keys" Oct 30 22:33:38.315565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315525 4292 cpu_manager.go:414] "containerMap: removing container" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" Oct 30 22:33:38.315565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315533 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" Oct 30 22:33:38.315869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315685 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5dc64bf-8a9b-405b-9826-bb5b155a03a4" containerName="revoke-keys" Oct 30 22:33:38.315869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315706 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="23078e7c-36a3-4bb0-a582-b72f4faad4b0" containerName="istio-proxy" Oct 30 22:33:38.315869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.315723 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c275a4-0c6f-469a-815d-d0cb228b5cc0" containerName="istio-proxy" Oct 30 22:33:38.320911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.320890 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.349983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.349954 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.350113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.350074 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf7d\" (UniqueName: \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.375673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.375649 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e54161\"" Oct 30 22:33:38.379083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.379053 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:33:38.489027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.485788 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.489027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.485911 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf7d\" (UniqueName: \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.489027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.486224 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:33:38.489027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.487432 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.492532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.492285 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.534686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.534654 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:33:38.549616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.549538 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf7d\" (UniqueName: \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\") pod \"payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.551697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.551674 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e54161-dockercfg-qxznd\"" Oct 30 22:33:38.635271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.635237 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:38.688503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.688292 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.688503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.688445 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnb9\" (UniqueName: \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.789902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.789822 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.790062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.789954 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnb9\" (UniqueName: \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.790692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.790675 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:38.852929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:38.852900 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnb9\" (UniqueName: \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\") pod \"payload-processing-e2e-inf-b-e54161-74d8458447-j45vx\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:39.013207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.013179 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:33:39.013766 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:33:39.013744 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc319bc3b_25e7_407a_9769_d4e6be68ad00.slice/crio-bb8e07a6ee25097984780b73390d3f689ae5659d140d80509574fa510e7284bb WatchSource:0}: Error finding container bb8e07a6ee25097984780b73390d3f689ae5659d140d80509574fa510e7284bb: Status 404 returned error can't find the container with id bb8e07a6ee25097984780b73390d3f689ae5659d140d80509574fa510e7284bb Oct 30 22:33:39.110959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.110929 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:39.111195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.111173 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:33:39.222850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.222816 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:33:39.245533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.245504 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" event={"ID":"c319bc3b-25e7-407a-9769-d4e6be68ad00","Type":"ContainerStarted","Data":"ec76d77a052a0900512f3fc71cc6b7b87d964c7b19c0cc48748c81fba68062ae"} Oct 30 22:33:39.245711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.245547 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" event={"ID":"c319bc3b-25e7-407a-9769-d4e6be68ad00","Type":"ContainerStarted","Data":"bb8e07a6ee25097984780b73390d3f689ae5659d140d80509574fa510e7284bb"} Oct 30 22:33:39.245711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.245599 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:39.473568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.473512 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" podStartSLOduration=1.473499809 podStartE2EDuration="1.473499809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:39.472435841 +0000 UTC m=+2347.441255406" watchObservedRunningTime="2026-10-30 22:33:39.473499809 +0000 UTC m=+2347.442319372" Oct 30 22:33:39.486443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:39.486419 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:33:39.486840 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:33:39.486817 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce95e8f_bf13_4dcf_a865_b3f472f6dc5a.slice/crio-fa8fa2ab1138518cad1aa0b0740610d34c50e89117ab302adf354757be38a5c3 WatchSource:0}: Error finding container fa8fa2ab1138518cad1aa0b0740610d34c50e89117ab302adf354757be38a5c3: Status 404 returned error can't find the container with id fa8fa2ab1138518cad1aa0b0740610d34c50e89117ab302adf354757be38a5c3 Oct 30 22:33:40.258887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:40.258835 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" event={"ID":"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a","Type":"ContainerStarted","Data":"8f4e3f33a94e5a24cdc64cf87e7cf442ee9ecc460bacd626d7b297ebc87c06d7"} Oct 30 22:33:40.259176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:40.258920 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" event={"ID":"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a","Type":"ContainerStarted","Data":"fa8fa2ab1138518cad1aa0b0740610d34c50e89117ab302adf354757be38a5c3"} Oct 30 22:33:40.259176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:40.259114 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:40.380125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:40.380081 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" podStartSLOduration=2.380067124 podStartE2EDuration="2.380067124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:40.379016855 +0000 UTC m=+2348.347836417" watchObservedRunningTime="2026-10-30 22:33:40.380067124 +0000 UTC m=+2348.348886689" Oct 30 22:33:44.112107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.112049 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" podUID="f0562400-f4ae-4961-9ee0-4670d5a16e90" containerName="payload-processing" containerID="cri-o://3647d5257689e9129433e8f6dfe99e7ec78837c1fc12a871e928bee9c37b9c1e" gracePeriod=25 Oct 30 22:33:44.223225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.223168 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" podUID="27599305-8e1a-452d-afb3-dd9a446ed3a8" containerName="payload-pre-processing" containerID="cri-o://cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef" gracePeriod=25 Oct 30 22:33:44.285771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.285734 4292 generic.go:356] "Generic (PLEG): container finished" podID="f0562400-f4ae-4961-9ee0-4670d5a16e90" containerID="3647d5257689e9129433e8f6dfe99e7ec78837c1fc12a871e928bee9c37b9c1e" exitCode=0 Oct 30 22:33:44.285952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.285775 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" event={"ID":"f0562400-f4ae-4961-9ee0-4670d5a16e90","Type":"ContainerDied","Data":"3647d5257689e9129433e8f6dfe99e7ec78837c1fc12a871e928bee9c37b9c1e"} Oct 30 22:33:44.376355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.376289 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:33:44.459953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.459918 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\") pod \"f0562400-f4ae-4961-9ee0-4670d5a16e90\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " Oct 30 22:33:44.460120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.460031 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\" (UniqueName: \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\") pod \"f0562400-f4ae-4961-9ee0-4670d5a16e90\" (UID: \"f0562400-f4ae-4961-9ee0-4670d5a16e90\") " Oct 30 22:33:44.460376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.460349 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume" pod "f0562400-f4ae-4961-9ee0-4670d5a16e90" (UID: "f0562400-f4ae-4961-9ee0-4670d5a16e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:44.460511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.460492 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0562400-f4ae-4961-9ee0-4670d5a16e90-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.463279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.463251 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw" pod "f0562400-f4ae-4961-9ee0-4670d5a16e90" (UID: "f0562400-f4ae-4961-9ee0-4670d5a16e90"). InnerVolumeSpecName "kube-api-access-fz8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:44.493454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.493434 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:33:44.561494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.561464 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\" (UniqueName: \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\") pod \"27599305-8e1a-452d-afb3-dd9a446ed3a8\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " Oct 30 22:33:44.561646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.561566 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\") pod \"27599305-8e1a-452d-afb3-dd9a446ed3a8\" (UID: \"27599305-8e1a-452d-afb3-dd9a446ed3a8\") " Oct 30 22:33:44.561874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.561855 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume" pod "27599305-8e1a-452d-afb3-dd9a446ed3a8" (UID: "27599305-8e1a-452d-afb3-dd9a446ed3a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:44.561937 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.561892 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8gw\" (UniqueName: \"kubernetes.io/projected/f0562400-f4ae-4961-9ee0-4670d5a16e90-kube-api-access-fz8gw\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.563622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.563598 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k" pod "27599305-8e1a-452d-afb3-dd9a446ed3a8" (UID: "27599305-8e1a-452d-afb3-dd9a446ed3a8"). InnerVolumeSpecName "kube-api-access-pgl5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:44.663393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.663328 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgl5k\" (UniqueName: \"kubernetes.io/projected/27599305-8e1a-452d-afb3-dd9a446ed3a8-kube-api-access-pgl5k\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.663393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:44.663351 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27599305-8e1a-452d-afb3-dd9a446ed3a8-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:45.293901 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.293864 4292 generic.go:356] "Generic (PLEG): container finished" podID="27599305-8e1a-452d-afb3-dd9a446ed3a8" containerID="cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef" exitCode=0 Oct 30 22:33:45.294289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.293905 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" Oct 30 22:33:45.294289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.293955 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" event={"ID":"27599305-8e1a-452d-afb3-dd9a446ed3a8","Type":"ContainerDied","Data":"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef"} Oct 30 22:33:45.294289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.294007 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk" event={"ID":"27599305-8e1a-452d-afb3-dd9a446ed3a8","Type":"ContainerDied","Data":"7157b6adcc029d7bfd03f600e1b3acdee827ec4571bdfb94e0b4c66f53e4156b"} Oct 30 22:33:45.294289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.294030 4292 scope.go:122] "RemoveContainer" containerID="cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef" Oct 30 22:33:45.295495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.295473 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" event={"ID":"f0562400-f4ae-4961-9ee0-4670d5a16e90","Type":"ContainerDied","Data":"a16e5a1342e4f17fff54fb56816856d00b704b07c3d10e1ac114c88a3c71a76a"} Oct 30 22:33:45.295615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.295522 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng" Oct 30 22:33:45.305454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.305431 4292 scope.go:122] "RemoveContainer" containerID="cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef" Oct 30 22:33:45.305773 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:45.305752 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef\": container with ID starting with cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef not found: ID does not exist" containerID="cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef" Oct 30 22:33:45.305847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.305779 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef"} err="failed to get container status \"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef\": rpc error: code = NotFound desc = could not find container \"cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef\": container with ID starting with cb0b923b19bde1e37069422d44e278b7ced62162658971cac084877285e719ef not found: ID does not exist" Oct 30 22:33:45.305847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.305795 4292 scope.go:122] "RemoveContainer" containerID="3647d5257689e9129433e8f6dfe99e7ec78837c1fc12a871e928bee9c37b9c1e" Oct 30 22:33:45.465035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.464964 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:33:45.515119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.515091 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18515933-69cb874d4f-7plng"] Oct 30 22:33:45.580084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.580057 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:33:45.585402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:45.585380 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18515933-6948bdcd8c-7hpnk"] Oct 30 22:33:46.146966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:46.146927 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27599305-8e1a-452d-afb3-dd9a446ed3a8" path="/var/lib/kubelet/pods/27599305-8e1a-452d-afb3-dd9a446ed3a8/volumes" Oct 30 22:33:46.147352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:46.147337 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0562400-f4ae-4961-9ee0-4670d5a16e90" path="/var/lib/kubelet/pods/f0562400-f4ae-4961-9ee0-4670d5a16e90/volumes" Oct 30 22:33:50.262802 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:50.262769 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:33:51.269122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:51.269087 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:33:56.639557 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:33:56.639521 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:33:59.400840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:59.400805 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj"] Oct 30 22:33:59.451292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:33:59.451261 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18515933-9nscj"] Oct 30 22:34:00.147180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:00.147148 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5dc64bf-8a9b-405b-9826-bb5b155a03a4" path="/var/lib/kubelet/pods/a5dc64bf-8a9b-405b-9826-bb5b155a03a4/volumes" Oct 30 22:34:09.806290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806253 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl"] Oct 30 22:34:09.806925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806852 4292 cpu_manager.go:414] "containerMap: removing container" podUID="27599305-8e1a-452d-afb3-dd9a446ed3a8" containerName="payload-pre-processing" Oct 30 22:34:09.806925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806865 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27599305-8e1a-452d-afb3-dd9a446ed3a8" containerName="payload-pre-processing" Oct 30 22:34:09.806925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806898 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f0562400-f4ae-4961-9ee0-4670d5a16e90" containerName="payload-processing" Oct 30 22:34:09.806925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806904 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0562400-f4ae-4961-9ee0-4670d5a16e90" containerName="payload-processing" Oct 30 22:34:09.807150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.806997 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="27599305-8e1a-452d-afb3-dd9a446ed3a8" containerName="payload-pre-processing" Oct 30 22:34:09.807150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.807012 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0562400-f4ae-4961-9ee0-4670d5a16e90" containerName="payload-processing" Oct 30 22:34:09.810661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.810643 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.833277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.833250 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzft\" (UniqueName: \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.833407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.833390 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.833464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.833452 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.869133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.869107 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl"] Oct 30 22:34:09.916090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.916058 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:09.916249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.916119 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:09.916249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.916130 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:34:09.916249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.916146 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:09.934476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.934453 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.934617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.934535 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzft\" (UniqueName: \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.934699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.934658 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.935254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.935234 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.936663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.936643 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:09.998608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:09.998568 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzft\" (UniqueName: \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\") pod \"maas-api-revoke-sub-823eee4e-25xxl\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:10.124641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:10.124561 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:10.503043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:10.503004 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl"] Oct 30 22:34:10.503188 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:34:10.503081 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ec5de6_476a_4202_8857_b009189837c5.slice/crio-deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e WatchSource:0}: Error finding container deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e: Status 404 returned error can't find the container with id deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e Oct 30 22:34:11.462063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:11.462024 4292 generic.go:356] "Generic (PLEG): container finished" podID="33ec5de6-476a-4202-8857-b009189837c5" containerID="aa384c0bf1d5494e7a3e1cadaffd35180f91179ad5eb7619b180ca93d420583f" exitCode=0 Oct 30 22:34:11.462394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:11.462105 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" event={"ID":"33ec5de6-476a-4202-8857-b009189837c5","Type":"ContainerDied","Data":"aa384c0bf1d5494e7a3e1cadaffd35180f91179ad5eb7619b180ca93d420583f"} Oct 30 22:34:11.462394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:11.462181 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" event={"ID":"33ec5de6-476a-4202-8857-b009189837c5","Type":"ContainerStarted","Data":"deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e"} Oct 30 22:34:12.617907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.617883 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:12.664397 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.664373 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\") pod \"33ec5de6-476a-4202-8857-b009189837c5\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " Oct 30 22:34:12.664514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.664485 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\") pod \"33ec5de6-476a-4202-8857-b009189837c5\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " Oct 30 22:34:12.664563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.664548 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\" (UniqueName: \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\") pod \"33ec5de6-476a-4202-8857-b009189837c5\" (UID: \"33ec5de6-476a-4202-8857-b009189837c5\") " Oct 30 22:34:12.664843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.664815 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca" pod "33ec5de6-476a-4202-8857-b009189837c5" (UID: "33ec5de6-476a-4202-8857-b009189837c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:12.666500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.666475 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca" pod "33ec5de6-476a-4202-8857-b009189837c5" (UID: "33ec5de6-476a-4202-8857-b009189837c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:12.666500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.666491 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft" pod "33ec5de6-476a-4202-8857-b009189837c5" (UID: "33ec5de6-476a-4202-8857-b009189837c5"). InnerVolumeSpecName "kube-api-access-hxzft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:12.766074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.765986 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec5de6-476a-4202-8857-b009189837c5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.766074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.766029 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxzft\" (UniqueName: \"kubernetes.io/projected/33ec5de6-476a-4202-8857-b009189837c5-kube-api-access-hxzft\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.766074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:12.766046 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec5de6-476a-4202-8857-b009189837c5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.476110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:13.476081 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" Oct 30 22:34:13.476286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:13.476075 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl" event={"ID":"33ec5de6-476a-4202-8857-b009189837c5","Type":"ContainerDied","Data":"deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e"} Oct 30 22:34:13.476286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:13.476212 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deaec72606574a50b98f5d786a1a7ecbf73a0b828876aa727acb2f6b88e3a29e" Oct 30 22:34:26.640860 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:34:26.640815 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:34:37.413224 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:37.413194 4292 scope.go:122] "RemoveContainer" containerID="33a6bdca3f74a37aa5b962126528aeb3993f28d8c0ef5cef5116d49f87a87da6" Oct 30 22:34:37.422931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:37.422910 4292 scope.go:122] "RemoveContainer" containerID="00061fbd25a0d7b5f29a0ef98f954c02db5984c8571c420d17b7eda0ec14679f" Oct 30 22:34:40.751751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.751718 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95"] Oct 30 22:34:40.752271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.752257 4292 cpu_manager.go:414] "containerMap: removing container" podUID="33ec5de6-476a-4202-8857-b009189837c5" containerName="revoke-keys" Oct 30 22:34:40.752318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.752271 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ec5de6-476a-4202-8857-b009189837c5" containerName="revoke-keys" Oct 30 22:34:40.752392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.752384 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ec5de6-476a-4202-8857-b009189837c5" containerName="revoke-keys" Oct 30 22:34:40.757322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.757305 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.810564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.810521 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp67\" (UniqueName: \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.810755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.810642 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.810755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.810703 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.816499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.816473 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95"] Oct 30 22:34:40.861968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.861935 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:40.862119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.862004 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:40.862119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.862002 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:40.862119 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.862048 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:34:40.912181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.912156 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.912330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.912314 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp67\" (UniqueName: \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.912407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.912348 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.913173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.913147 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.914716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.914695 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:40.975808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:40.975782 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp67\" (UniqueName: \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\") pod \"maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:41.070749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:41.070723 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:41.448440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:41.448402 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95"] Oct 30 22:34:41.691318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:41.691283 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" event={"ID":"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d","Type":"ContainerStarted","Data":"ff66bf5d80213a42098d6522e375cce65a27364240e51f98e6dee83cf43fc358"} Oct 30 22:34:42.699483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:42.699444 4292 generic.go:356] "Generic (PLEG): container finished" podID="65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" containerID="18cb15e231156704c2c70686f8cbba4f81d58e5713234718cd30528861b81574" exitCode=0 Oct 30 22:34:42.699984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:42.699530 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" event={"ID":"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d","Type":"ContainerDied","Data":"18cb15e231156704c2c70686f8cbba4f81d58e5713234718cd30528861b81574"} Oct 30 22:34:43.852858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.852834 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:43.957125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.957064 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\") pod \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " Oct 30 22:34:43.957243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.957175 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\" (UniqueName: \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\") pod \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " Oct 30 22:34:43.957298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.957286 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\") pod \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\" (UID: \"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d\") " Oct 30 22:34:43.957460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.957433 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca" pod "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" (UID: "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:43.957677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.957653 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.959259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.959238 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67" pod "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" (UID: "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d"). InnerVolumeSpecName "kube-api-access-xfp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.959518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:43.959288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca" pod "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" (UID: "65ef73aa-1aff-4a8f-8f5c-0b2e0888215d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:44.058426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:44.058395 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfp67\" (UniqueName: \"kubernetes.io/projected/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-kube-api-access-xfp67\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:44.058426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:44.058423 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:44.714411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:44.714376 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" event={"ID":"65ef73aa-1aff-4a8f-8f5c-0b2e0888215d","Type":"ContainerDied","Data":"ff66bf5d80213a42098d6522e375cce65a27364240e51f98e6dee83cf43fc358"} Oct 30 22:34:44.714633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:44.714437 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff66bf5d80213a42098d6522e375cce65a27364240e51f98e6dee83cf43fc358" Oct 30 22:34:44.714633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:44.714383 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95" Oct 30 22:34:45.125464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:45.125431 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:34:45.144066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:45.144037 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:34:51.014512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:51.014474 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:34:51.014966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:51.014706 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" containerID="cri-o://6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2" gracePeriod=30 Oct 30 22:34:51.658479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:51.658439 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:34:55.114522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.114486 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc"] Oct 30 22:34:55.115114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.115099 4292 cpu_manager.go:414] "containerMap: removing container" podUID="65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" containerName="revoke-keys" Oct 30 22:34:55.115162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.115115 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" containerName="revoke-keys" Oct 30 22:34:55.115222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.115213 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" containerName="revoke-keys" Oct 30 22:34:55.118745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.118727 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.177181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.177157 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p"] Oct 30 22:34:55.180916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.180901 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.187485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.187455 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.187610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.187587 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5g2\" (UniqueName: \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.187723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.187696 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.224455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.224425 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:55.224569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.224452 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:55.225308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.225279 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:55.225308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.225284 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:34:55.229812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.229789 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc"] Oct 30 22:34:55.281379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.281357 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p"] Oct 30 22:34:55.288318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288295 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk686\" (UniqueName: \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.288444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288365 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.288513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288459 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5g2\" (UniqueName: \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.288513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288498 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.288638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288531 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.288693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.288663 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.289467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.289446 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.290731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.290709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.352302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.352280 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5g2\" (UniqueName: \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\") pod \"maas-api-revoke-sub-a9c4acb8-6qdwc\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.389488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.389430 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk686\" (UniqueName: \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.389622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.389547 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.389622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.389606 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.390198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.390174 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.391566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.391548 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.432446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.432421 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:55.453852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.453826 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk686\" (UniqueName: \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\") pod \"maas-api-revoke-sub-a69fa0f9-5tp5p\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.493929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.493898 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:55.821777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.820852 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc"] Oct 30 22:34:55.823721 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:34:55.823697 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568f2838_6128_4372_96e8_b1283305a48d.slice/crio-5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89 WatchSource:0}: Error finding container 5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89: Status 404 returned error can't find the container with id 5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89 Oct 30 22:34:55.927618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:55.927594 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p"] Oct 30 22:34:55.928024 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:34:55.927997 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2483e519_df79_4c28_ab9b_597061dd78ea.slice/crio-a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615 WatchSource:0}: Error finding container a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615: Status 404 returned error can't find the container with id a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615 Oct 30 22:34:56.269098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.269071 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:34:56.402508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402442 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402505 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402629 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402681 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402762 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402800 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.402995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402811 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:56.402995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402879 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:56.402995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.402887 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.403167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403030 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:56.403167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403066 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.403167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403123 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\") pod \"4cb7cda7-c269-461a-ac65-2c400eec3374\" (UID: \"4cb7cda7-c269-461a-ac65-2c400eec3374\") " Oct 30 22:34:56.403338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403068 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:56.403338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403235 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:56.403672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403643 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.403672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403672 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.403867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403687 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.403867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403705 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb7cda7-c269-461a-ac65-2c400eec3374-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.403867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.403720 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.405683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.405658 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:56.405780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.405693 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:56.405995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.405963 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "kube-api-access-2r29v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:56.406111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.406064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token" pod "4cb7cda7-c269-461a-ac65-2c400eec3374" (UID: "4cb7cda7-c269-461a-ac65-2c400eec3374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:56.504706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.504679 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.504706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.504704 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r29v\" (UniqueName: \"kubernetes.io/projected/4cb7cda7-c269-461a-ac65-2c400eec3374-kube-api-access-2r29v\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.504879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.504716 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.504879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.504727 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb7cda7-c269-461a-ac65-2c400eec3374-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.639639 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:34:56.639599 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:34:56.659874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.659791 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" containerID="cri-o://969468c3c5d00ea3402725a15d9fa11bc6edc47d75c3ae33330bca99ad4df64a" gracePeriod=25 Oct 30 22:34:56.794325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.794281 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" event={"ID":"2483e519-df79-4c28-ab9b-597061dd78ea","Type":"ContainerStarted","Data":"2838ddffe2194a77a9b990e54bdff176c84045603d1bcae24cafdb46cc5287d2"} Oct 30 22:34:56.794464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.794358 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" event={"ID":"2483e519-df79-4c28-ab9b-597061dd78ea","Type":"ContainerStarted","Data":"a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615"} Oct 30 22:34:56.796361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.796332 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" event={"ID":"568f2838-6128-4372-96e8-b1283305a48d","Type":"ContainerStarted","Data":"e728e8a591eb78571a1b85dafd193a8a0166ad8891890ad5e768343dc979352d"} Oct 30 22:34:56.796515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.796387 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" event={"ID":"568f2838-6128-4372-96e8-b1283305a48d","Type":"ContainerStarted","Data":"5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89"} Oct 30 22:34:56.798319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.798291 4292 generic.go:356] "Generic (PLEG): container finished" podID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerID="6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2" exitCode=0 Oct 30 22:34:56.798446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.798429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerDied","Data":"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2"} Oct 30 22:34:56.798520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.798479 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" event={"ID":"4cb7cda7-c269-461a-ac65-2c400eec3374","Type":"ContainerDied","Data":"6b24124e9c8f41cbd2507d25fab15f996142dc061a05441a6a1ec298cdd8a82e"} Oct 30 22:34:56.798520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.798515 4292 scope.go:122] "RemoveContainer" containerID="6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2" Oct 30 22:34:56.798704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.798689 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm" Oct 30 22:34:56.847935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.847918 4292 scope.go:122] "RemoveContainer" containerID="6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f" Oct 30 22:34:56.915711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.915643 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" podStartSLOduration=1.9156291909999998 podStartE2EDuration="1.915629191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:56.913211299 +0000 UTC m=+2424.882030866" watchObservedRunningTime="2026-10-30 22:34:56.915629191 +0000 UTC m=+2424.884448754" Oct 30 22:34:56.920980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.920959 4292 scope.go:122] "RemoveContainer" containerID="6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2" Oct 30 22:34:56.921272 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:34:56.921250 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2\": container with ID starting with 6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2 not found: ID does not exist" containerID="6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2" Oct 30 22:34:56.921329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.921283 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2"} err="failed to get container status \"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2\": rpc error: code = NotFound desc = could not find container \"6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2\": container with ID starting with 6f80e1db6365216d6e10584bd170d3ddb7e7c8d0d769dd319983ddcf65e140c2 not found: ID does not exist" Oct 30 22:34:56.921329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.921305 4292 scope.go:122] "RemoveContainer" containerID="6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f" Oct 30 22:34:56.921606 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:34:56.921565 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f\": container with ID starting with 6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f not found: ID does not exist" containerID="6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f" Oct 30 22:34:56.921659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:56.921617 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f"} err="failed to get container status \"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f\": rpc error: code = NotFound desc = could not find container \"6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f\": container with ID starting with 6df06e47571bfc47bc6a763af4f6fb3040bd97b68806d7ed961c52af099de47f not found: ID does not exist" Oct 30 22:34:57.152928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.152900 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:34:57.158302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.158273 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18515933-openshift-default-57bc477dfc-xxcxm"] Oct 30 22:34:57.808570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.808536 4292 generic.go:356] "Generic (PLEG): container finished" podID="2483e519-df79-4c28-ab9b-597061dd78ea" containerID="2838ddffe2194a77a9b990e54bdff176c84045603d1bcae24cafdb46cc5287d2" exitCode=0 Oct 30 22:34:57.809133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.808607 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" event={"ID":"2483e519-df79-4c28-ab9b-597061dd78ea","Type":"ContainerDied","Data":"2838ddffe2194a77a9b990e54bdff176c84045603d1bcae24cafdb46cc5287d2"} Oct 30 22:34:57.810429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.810401 4292 generic.go:356] "Generic (PLEG): container finished" podID="568f2838-6128-4372-96e8-b1283305a48d" containerID="e728e8a591eb78571a1b85dafd193a8a0166ad8891890ad5e768343dc979352d" exitCode=0 Oct 30 22:34:57.810540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.810486 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" event={"ID":"568f2838-6128-4372-96e8-b1283305a48d","Type":"ContainerDied","Data":"e728e8a591eb78571a1b85dafd193a8a0166ad8891890ad5e768343dc979352d"} Oct 30 22:34:57.969115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:57.969090 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:58.020352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.020325 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\") pod \"568f2838-6128-4372-96e8-b1283305a48d\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " Oct 30 22:34:58.020527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.020390 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\") pod \"568f2838-6128-4372-96e8-b1283305a48d\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " Oct 30 22:34:58.020645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.020628 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\" (UniqueName: \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\") pod \"568f2838-6128-4372-96e8-b1283305a48d\" (UID: \"568f2838-6128-4372-96e8-b1283305a48d\") " Oct 30 22:34:58.020763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.020738 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca" pod "568f2838-6128-4372-96e8-b1283305a48d" (UID: "568f2838-6128-4372-96e8-b1283305a48d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:58.021043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.021026 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568f2838-6128-4372-96e8-b1283305a48d-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:58.022387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.022353 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca" pod "568f2838-6128-4372-96e8-b1283305a48d" (UID: "568f2838-6128-4372-96e8-b1283305a48d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:58.022493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.022438 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2" pod "568f2838-6128-4372-96e8-b1283305a48d" (UID: "568f2838-6128-4372-96e8-b1283305a48d"). InnerVolumeSpecName "kube-api-access-wc5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:58.121930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.121849 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5g2\" (UniqueName: \"kubernetes.io/projected/568f2838-6128-4372-96e8-b1283305a48d-kube-api-access-wc5g2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:58.121930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.121885 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568f2838-6128-4372-96e8-b1283305a48d-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:58.147132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.147101 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" path="/var/lib/kubelet/pods/4cb7cda7-c269-461a-ac65-2c400eec3374/volumes" Oct 30 22:34:58.321192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.321155 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:58.819686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.819635 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" event={"ID":"568f2838-6128-4372-96e8-b1283305a48d","Type":"ContainerDied","Data":"5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89"} Oct 30 22:34:58.820144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.819742 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b1342d1af19373684dfc8f4e46141d85b68464993992dce134d3518188a5a89" Oct 30 22:34:58.820144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.819649 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc" Oct 30 22:34:58.954056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:58.954029 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:59.031121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.031095 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\") pod \"2483e519-df79-4c28-ab9b-597061dd78ea\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " Oct 30 22:34:59.031252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.031156 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\" (UniqueName: \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\") pod \"2483e519-df79-4c28-ab9b-597061dd78ea\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " Oct 30 22:34:59.031296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.031253 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\") pod \"2483e519-df79-4c28-ab9b-597061dd78ea\" (UID: \"2483e519-df79-4c28-ab9b-597061dd78ea\") " Oct 30 22:34:59.031440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.031415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca" pod "2483e519-df79-4c28-ab9b-597061dd78ea" (UID: "2483e519-df79-4c28-ab9b-597061dd78ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:59.031566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.031547 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:59.033259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.033239 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca" pod "2483e519-df79-4c28-ab9b-597061dd78ea" (UID: "2483e519-df79-4c28-ab9b-597061dd78ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:59.033259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.033252 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686" pod "2483e519-df79-4c28-ab9b-597061dd78ea" (UID: "2483e519-df79-4c28-ab9b-597061dd78ea"). InnerVolumeSpecName "kube-api-access-rk686". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:59.132826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.132755 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk686\" (UniqueName: \"kubernetes.io/projected/2483e519-df79-4c28-ab9b-597061dd78ea-kube-api-access-rk686\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:59.132826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.132782 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2483e519-df79-4c28-ab9b-597061dd78ea-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:59.826913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.826885 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" Oct 30 22:34:59.827325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.826874 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p" event={"ID":"2483e519-df79-4c28-ab9b-597061dd78ea","Type":"ContainerDied","Data":"a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615"} Oct 30 22:34:59.827325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:34:59.827016 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6066a747a2aab6eaf7950f594703310b7e8404e724451a474e157035b7a9615" Oct 30 22:35:00.148283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.148196 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" containerID="cri-o://35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d" gracePeriod=45 Oct 30 22:35:00.163422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.163390 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" containerID="cri-o://f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480" gracePeriod=45 Oct 30 22:35:00.451216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.451187 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:35:00.470859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.470836 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:35:00.548567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548537 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.548762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548654 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.548762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548706 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.548762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548723 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.548762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548741 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548760 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\" (UniqueName: \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548789 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548805 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548829 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\" (UniqueName: \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.548993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548986 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\") pod \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\" (UID: \"062d5db4-1a5c-44b8-af2d-005e0a54cf5c\") " Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.548995 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549014 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549004 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549095 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\") pod \"5c755482-36fa-474b-a909-a71616315b3c\" (UID: \"5c755482-36fa-474b-a909-a71616315b3c\") " Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549207 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.549358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549348 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.549793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549759 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.549960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549781 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.550068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549974 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.550068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.549993 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.550068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.550007 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.550068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.550021 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.551329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551301 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.551448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551405 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:00.551520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551484 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:00.551614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551590 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:00.551670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551616 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m" pod "5c755482-36fa-474b-a909-a71616315b3c" (UID: "5c755482-36fa-474b-a909-a71616315b3c"). InnerVolumeSpecName "kube-api-access-trz7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:00.551903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.551887 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb" pod "062d5db4-1a5c-44b8-af2d-005e0a54cf5c" (UID: "062d5db4-1a5c-44b8-af2d-005e0a54cf5c"). InnerVolumeSpecName "kube-api-access-sffhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:00.559342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559317 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:35:00.559899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559881 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559899 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559934 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559940 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559948 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559953 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559963 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2483e519-df79-4c28-ab9b-597061dd78ea" containerName="revoke-keys" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559968 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2483e519-df79-4c28-ab9b-597061dd78ea" containerName="revoke-keys" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559978 4292 cpu_manager.go:414] "containerMap: removing container" podUID="568f2838-6128-4372-96e8-b1283305a48d" containerName="revoke-keys" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559985 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568f2838-6128-4372-96e8-b1283305a48d" containerName="revoke-keys" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.559999 4292 cpu_manager.go:414] "containerMap: removing container" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" Oct 30 22:35:00.560023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560004 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560087 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560096 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c755482-36fa-474b-a909-a71616315b3c" containerName="main" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560104 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2483e519-df79-4c28-ab9b-597061dd78ea" containerName="revoke-keys" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560110 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="568f2838-6128-4372-96e8-b1283305a48d" containerName="revoke-keys" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560118 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" Oct 30 22:35:00.560464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.560127 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb7cda7-c269-461a-ac65-2c400eec3374" containerName="istio-proxy" Oct 30 22:35:00.563688 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.563672 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.629785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.629752 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:35:00.651703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.651670 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.651857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.651722 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.651924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.651857 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.651981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.651960 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.651997 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652030 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652178 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652323 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652414 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2sc\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.652544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652530 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c755482-36fa-474b-a909-a71616315b3c-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652551 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652566 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffhb\" (UniqueName: \"kubernetes.io/projected/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-kube-api-access-sffhb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652610 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652626 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trz7m\" (UniqueName: \"kubernetes.io/projected/5c755482-36fa-474b-a909-a71616315b3c-kube-api-access-trz7m\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652640 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062d5db4-1a5c-44b8-af2d-005e0a54cf5c-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.652750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.652654 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c755482-36fa-474b-a909-a71616315b3c-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:00.694667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.694562 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e54161-openshift-default-dockercfg-dc5xx\"" Oct 30 22:35:00.753783 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.753744 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.753963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.753807 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.753963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.753867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.753963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.753947 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.753983 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754027 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754072 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754156 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754200 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2sc\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754205 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754215 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754292 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754523 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.754927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.754904 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.756345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.756323 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.756444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.756379 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.822042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.822007 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.823559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.823535 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2sc\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\") pod \"e2e-inf-b-e54161-openshift-default-7899496757-prk5m\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:00.834904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.834879 4292 generic.go:356] "Generic (PLEG): container finished" podID="5c755482-36fa-474b-a909-a71616315b3c" containerID="f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480" exitCode=0 Oct 30 22:35:00.835277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.834918 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" Oct 30 22:35:00.835277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.834963 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" event={"ID":"5c755482-36fa-474b-a909-a71616315b3c","Type":"ContainerDied","Data":"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480"} Oct 30 22:35:00.835277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.835010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j" event={"ID":"5c755482-36fa-474b-a909-a71616315b3c","Type":"ContainerDied","Data":"e68ad46cb22b5cda65a97ae848e24d0954a14e7deec882beede95036d1a1ef15"} Oct 30 22:35:00.835277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.835030 4292 scope.go:122] "RemoveContainer" containerID="f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480" Oct 30 22:35:00.836539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.836513 4292 generic.go:356] "Generic (PLEG): container finished" podID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerID="35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d" exitCode=0 Oct 30 22:35:00.836649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.836544 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" event={"ID":"062d5db4-1a5c-44b8-af2d-005e0a54cf5c","Type":"ContainerDied","Data":"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d"} Oct 30 22:35:00.836649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.836565 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" Oct 30 22:35:00.836649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.836607 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" event={"ID":"062d5db4-1a5c-44b8-af2d-005e0a54cf5c","Type":"ContainerDied","Data":"d1f886b06a282d491f391a78c1a08b23dff6bf2ea46fd6972a2b65b223916dfa"} Oct 30 22:35:00.845292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.845277 4292 scope.go:122] "RemoveContainer" containerID="f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480" Oct 30 22:35:00.845586 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:35:00.845557 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480\": container with ID starting with f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480 not found: ID does not exist" containerID="f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480" Oct 30 22:35:00.845666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.845603 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480"} err="failed to get container status \"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480\": rpc error: code = NotFound desc = could not find container \"f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480\": container with ID starting with f4d7d3c4ebfd3ef1fb19538100e59a73f63fdc19f406a8bc64033475689ed480 not found: ID does not exist" Oct 30 22:35:00.845666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.845626 4292 scope.go:122] "RemoveContainer" containerID="35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d" Oct 30 22:35:00.857156 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.857136 4292 scope.go:122] "RemoveContainer" containerID="35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d" Oct 30 22:35:00.857398 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:35:00.857379 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d\": container with ID starting with 35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d not found: ID does not exist" containerID="35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d" Oct 30 22:35:00.857452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.857405 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d"} err="failed to get container status \"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d\": rpc error: code = NotFound desc = could not find container \"35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d\": container with ID starting with 35f642902f16b124ca5517bf71a42405d4a249baf186af3a694ff83470ecd53d not found: ID does not exist" Oct 30 22:35:00.879528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:00.879505 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:01.005630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.005547 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:35:01.011623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.011596 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/premium-w5-075cbb-kserve-69d4b5f686-zqv8j"] Oct 30 22:35:01.124570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.124537 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:35:01.131961 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.131932 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f"] Oct 30 22:35:01.308852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.308823 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:35:01.308968 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:35:01.308822 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163dfadf_f06d_44bf_84f7_db14adcfbb82.slice/crio-185f05a9b92a63f89054cef115134d5a21e2db6c392192ab7e09f11aabd92a5d WatchSource:0}: Error finding container 185f05a9b92a63f89054cef115134d5a21e2db6c392192ab7e09f11aabd92a5d: Status 404 returned error can't find the container with id 185f05a9b92a63f89054cef115134d5a21e2db6c392192ab7e09f11aabd92a5d Oct 30 22:35:01.311364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.311332 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:35:01.311479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.311410 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:35:01.311479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.311454 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:35:01.442473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.442425 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-075cbb/distinct-2-w5-075cbb-kserve-f6685668b-jd97f" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:01.847871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.847834 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" event={"ID":"163dfadf-f06d-44bf-84f7-db14adcfbb82","Type":"ContainerStarted","Data":"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6"} Oct 30 22:35:01.848271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.847911 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" event={"ID":"163dfadf-f06d-44bf-84f7-db14adcfbb82","Type":"ContainerStarted","Data":"185f05a9b92a63f89054cef115134d5a21e2db6c392192ab7e09f11aabd92a5d"} Oct 30 22:35:01.879686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.879647 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:01.973645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:01.973593 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podStartSLOduration=1.973563007 podStartE2EDuration="1.973563007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:01.972018778 +0000 UTC m=+2429.940838341" watchObservedRunningTime="2026-10-30 22:35:01.973563007 +0000 UTC m=+2429.942382645" Oct 30 22:35:02.146533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:02.146453 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062d5db4-1a5c-44b8-af2d-005e0a54cf5c" path="/var/lib/kubelet/pods/062d5db4-1a5c-44b8-af2d-005e0a54cf5c/volumes" Oct 30 22:35:02.146978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:02.146959 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c755482-36fa-474b-a909-a71616315b3c" path="/var/lib/kubelet/pods/5c755482-36fa-474b-a909-a71616315b3c/volumes" Oct 30 22:35:02.881808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:02.881775 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:35:02.882207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:02.881827 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 22:35:03.880315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:03.880282 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:35:03.880502 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:03.880336 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 22:35:04.880293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:04.880257 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:35:04.880717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:04.880306 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 22:35:05.880739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:05.880703 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:35:05.881158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:05.880758 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 22:35:06.880504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:06.880462 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:35:06.880733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:06.880515 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 22:35:07.885192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:07.885158 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:07.885621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:07.885602 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:07.886544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:07.886523 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:35:08.059049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:08.059014 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k"] Oct 30 22:35:08.116406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:08.116369 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aa98a1-6k65k"] Oct 30 22:35:08.148149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:08.148070 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fdea95-a4da-4d8f-a70e-504b5202bc0a" path="/var/lib/kubelet/pods/09fdea95-a4da-4d8f-a70e-504b5202bc0a/volumes" Oct 30 22:35:08.321506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:08.321464 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:16.958215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:16.958115 4292 generic.go:356] "Generic (PLEG): container finished" podID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerID="969468c3c5d00ea3402725a15d9fa11bc6edc47d75c3ae33330bca99ad4df64a" exitCode=0 Oct 30 22:35:16.958215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:16.958147 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" event={"ID":"3f3486b1-926d-4d61-9eae-7d62a3179d85","Type":"ContainerDied","Data":"969468c3c5d00ea3402725a15d9fa11bc6edc47d75c3ae33330bca99ad4df64a"} Oct 30 22:35:16.990587 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:16.990553 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:35:17.079510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.078632 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\" (UniqueName: \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\") pod \"3f3486b1-926d-4d61-9eae-7d62a3179d85\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " Oct 30 22:35:17.079510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.078722 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\") pod \"3f3486b1-926d-4d61-9eae-7d62a3179d85\" (UID: \"3f3486b1-926d-4d61-9eae-7d62a3179d85\") " Oct 30 22:35:17.080087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.080049 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume" pod "3f3486b1-926d-4d61-9eae-7d62a3179d85" (UID: "3f3486b1-926d-4d61-9eae-7d62a3179d85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:17.083030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.082993 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w" pod "3f3486b1-926d-4d61-9eae-7d62a3179d85" (UID: "3f3486b1-926d-4d61-9eae-7d62a3179d85"). InnerVolumeSpecName "kube-api-access-k4p5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:17.272946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.268006 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4p5w\" (UniqueName: \"kubernetes.io/projected/3f3486b1-926d-4d61-9eae-7d62a3179d85-kube-api-access-k4p5w\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:17.272946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.272839 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3486b1-926d-4d61-9eae-7d62a3179d85-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:17.979345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.979303 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" event={"ID":"3f3486b1-926d-4d61-9eae-7d62a3179d85","Type":"ContainerDied","Data":"52858bdbb916ddb2b174e00320bfc0e06575e12f8393556dd64d1e756d096fb4"} Oct 30 22:35:17.979978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.979957 4292 scope.go:122] "RemoveContainer" containerID="969468c3c5d00ea3402725a15d9fa11bc6edc47d75c3ae33330bca99ad4df64a" Oct 30 22:35:17.980246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:17.980227 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt" Oct 30 22:35:18.165708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.165667 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5xmvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:35:18.165994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.165963 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:18.194729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.194695 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:35:18.248645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.238487 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-075cbb-dc9b5fcff-vnfrt"] Oct 30 22:35:18.452730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.452693 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-slvnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:35:18.453070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:18.453020 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:20.147127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:20.147095 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" path="/var/lib/kubelet/pods/3f3486b1-926d-4d61-9eae-7d62a3179d85/volumes" Oct 30 22:35:24.058709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.058670 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f"] Oct 30 22:35:24.108955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.108927 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7"] Oct 30 22:35:24.111863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.111840 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1a8e0c-9hq5f"] Oct 30 22:35:24.114293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.114274 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9cb1d4-c2hz7"] Oct 30 22:35:24.146791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.146755 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8605ca55-a3b0-4466-8d3d-a56182d9a13a" path="/var/lib/kubelet/pods/8605ca55-a3b0-4466-8d3d-a56182d9a13a/volumes" Oct 30 22:35:24.147284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:24.147263 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75f6cc3-3526-42c7-a48a-3f8aedb43017" path="/var/lib/kubelet/pods/e75f6cc3-3526-42c7-a48a-3f8aedb43017/volumes" Oct 30 22:35:26.640780 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:35:26.640744 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:35:27.210669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.210637 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:35:27.211204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.211189 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" Oct 30 22:35:27.211257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.211205 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" Oct 30 22:35:27.211325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.211316 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3486b1-926d-4d61-9eae-7d62a3179d85" containerName="payload-pre-processing" Oct 30 22:35:27.216268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.216253 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.321212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.321178 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:35:27.321412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.321320 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd5407\"/\"default-dockercfg-l6z2p\"" Oct 30 22:35:27.321483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.321470 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd5407\"/\"openshift-service-ca.crt\"" Oct 30 22:35:27.321527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.321471 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd5407\"/\"kube-root-ca.crt\"" Oct 30 22:35:27.321527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.321503 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd5407\"/\"test-model-cd5407-kserve-self-signed-certs\"" Oct 30 22:35:27.325024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.324996 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.325159 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.325074 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.325220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.325200 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.325264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.325249 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.325311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.325281 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.325362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.325345 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8vg\" (UniqueName: \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.426706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.426668 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.426909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.426767 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.426909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.426815 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.426909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.426851 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.427098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.426973 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8vg\" (UniqueName: \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.427098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.427041 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.427322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.427297 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.427406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.427325 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.427449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.427423 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.429077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.429061 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.429290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.429274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.489938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.489876 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8vg\" (UniqueName: \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\") pod \"test-model-cd5407-kserve-599df5f9fc-pr9q4\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.528516 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.528493 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:27.914983 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:27.914945 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:35:27.915849 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:35:27.915816 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74c646e7_2555_4dcf_acba_6dda556dd948.slice/crio-a0a94291de4a9bdbfde7b68b5af88eae0866538495b58522d086b2e310ccc08a WatchSource:0}: Error finding container a0a94291de4a9bdbfde7b68b5af88eae0866538495b58522d086b2e310ccc08a: Status 404 returned error can't find the container with id a0a94291de4a9bdbfde7b68b5af88eae0866538495b58522d086b2e310ccc08a Oct 30 22:35:28.050016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:28.049974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" event={"ID":"74c646e7-2555-4dcf-acba-6dda556dd948","Type":"ContainerStarted","Data":"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb"} Oct 30 22:35:28.050172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:28.050024 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" event={"ID":"74c646e7-2555-4dcf-acba-6dda556dd948","Type":"ContainerStarted","Data":"a0a94291de4a9bdbfde7b68b5af88eae0866538495b58522d086b2e310ccc08a"} Oct 30 22:35:28.170770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:28.170678 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" podStartSLOduration=1.170665424 podStartE2EDuration="1.170665424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:28.169754981 +0000 UTC m=+2456.138574546" watchObservedRunningTime="2026-10-30 22:35:28.170665424 +0000 UTC m=+2456.139484989" Oct 30 22:35:28.529601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:28.529504 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:37.464033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:37.464001 4292 scope.go:122] "RemoveContainer" containerID="e70b6accf688eed37595861e4cdd654559859b74d1607b30e3277fb80d516ca8" Oct 30 22:35:37.529360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:37.529334 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:37.542243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:37.542215 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:38.127228 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:38.127194 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:35:43.784988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:43.784948 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95"] Oct 30 22:35:43.835441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:43.835408 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-075cbb-xvk95"] Oct 30 22:35:44.146995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:44.146963 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ef73aa-1aff-4a8f-8f5c-0b2e0888215d" path="/var/lib/kubelet/pods/65ef73aa-1aff-4a8f-8f5c-0b2e0888215d/volumes" Oct 30 22:35:51.472547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.472469 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk"] Oct 30 22:35:51.478631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.478601 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.535534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.535506 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:51.583445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.583415 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:51.583670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.583543 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:51.583670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.583606 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:35:51.588854 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.588817 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk"] Oct 30 22:35:51.600379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.600358 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.600504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.600397 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmj6\" (UniqueName: \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.600504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.600464 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.701680 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.701648 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.701874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.701695 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmj6\" (UniqueName: \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.701874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.701727 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.702450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.702423 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.704130 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.704101 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.765829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.765760 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmj6\" (UniqueName: \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.791743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.791721 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:51.869181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.868507 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:35:51.869181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:51.869020 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" containerID="cri-o://63562757b4f9e233a4d6bcb02a261f5298ba45d2c4420cf70e7ff0bcfc4b22ce" gracePeriod=30 Oct 30 22:35:52.198979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:52.198943 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk"] Oct 30 22:35:52.199227 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:35:52.199199 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4306e3ba_778d_4c3e_a10a_b75784768391.slice/crio-6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468 WatchSource:0}: Error finding container 6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468: Status 404 returned error can't find the container with id 6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468 Oct 30 22:35:53.211320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:53.211287 4292 generic.go:356] "Generic (PLEG): container finished" podID="4306e3ba-778d-4c3e-a10a-b75784768391" containerID="c32df4bf6a73e998afb891a92ac17e9e422c16897521a563937b9a54e6de748b" exitCode=0 Oct 30 22:35:53.211706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:53.211376 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" event={"ID":"4306e3ba-778d-4c3e-a10a-b75784768391","Type":"ContainerDied","Data":"c32df4bf6a73e998afb891a92ac17e9e422c16897521a563937b9a54e6de748b"} Oct 30 22:35:53.211706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:53.211429 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" event={"ID":"4306e3ba-778d-4c3e-a10a-b75784768391","Type":"ContainerStarted","Data":"6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468"} Oct 30 22:35:54.382930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.382903 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:54.448192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.448169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\") pod \"4306e3ba-778d-4c3e-a10a-b75784768391\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " Oct 30 22:35:54.448300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.448234 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\" (UniqueName: \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\") pod \"4306e3ba-778d-4c3e-a10a-b75784768391\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " Oct 30 22:35:54.448300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.448285 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\") pod \"4306e3ba-778d-4c3e-a10a-b75784768391\" (UID: \"4306e3ba-778d-4c3e-a10a-b75784768391\") " Oct 30 22:35:54.448713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.448685 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca" pod "4306e3ba-778d-4c3e-a10a-b75784768391" (UID: "4306e3ba-778d-4c3e-a10a-b75784768391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:54.450230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.450204 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca" pod "4306e3ba-778d-4c3e-a10a-b75784768391" (UID: "4306e3ba-778d-4c3e-a10a-b75784768391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:54.450533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.450507 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6" pod "4306e3ba-778d-4c3e-a10a-b75784768391" (UID: "4306e3ba-778d-4c3e-a10a-b75784768391"). InnerVolumeSpecName "kube-api-access-dhmj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:54.549953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.549920 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:54.549953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.549953 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmj6\" (UniqueName: \"kubernetes.io/projected/4306e3ba-778d-4c3e-a10a-b75784768391-kube-api-access-dhmj6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:54.550148 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:54.549968 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4306e3ba-778d-4c3e-a10a-b75784768391-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:55.235183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.235145 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" event={"ID":"4306e3ba-778d-4c3e-a10a-b75784768391","Type":"ContainerDied","Data":"6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468"} Oct 30 22:35:55.235358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.235193 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk" Oct 30 22:35:55.235358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.235220 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2c7545a3fc8f9adcf457821285ed29e0c7142a8de7da43d621f1bc48220468" Oct 30 22:35:55.250093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.250070 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:35:55.250597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.250568 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4306e3ba-778d-4c3e-a10a-b75784768391" containerName="revoke-keys" Oct 30 22:35:55.250655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.250599 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4306e3ba-778d-4c3e-a10a-b75784768391" containerName="revoke-keys" Oct 30 22:35:55.250702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.250693 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4306e3ba-778d-4c3e-a10a-b75784768391" containerName="revoke-keys" Oct 30 22:35:55.254016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.254001 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.312433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.312405 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e54161\"/\"kube-root-ca.crt\"" Oct 30 22:35:55.348440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.348418 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e54161\"/\"openshift-service-ca.crt\"" Oct 30 22:35:55.348621 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.348532 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e54161\"/\"test-model-e54161-kserve-self-signed-certs\"" Oct 30 22:35:55.348678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.348642 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e54161\"/\"default-dockercfg-rckct\"" Oct 30 22:35:55.351820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.351799 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:35:55.360181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360161 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.360272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360195 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.360272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360236 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.360387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360368 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.360461 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360445 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.360534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.360518 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszrg\" (UniqueName: \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461185 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461225 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461332 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461418 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461477 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszrg\" (UniqueName: \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461737 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461752 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.461924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.461827 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.463477 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.463453 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.463807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.463788 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.524740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.524684 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszrg\" (UniqueName: \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\") pod \"test-model-e54161-kserve-5bf748544c-5xcfn\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.568504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.568460 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:55.864413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.864379 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:35:55.979274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:55.979244 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:35:55.980110 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:35:55.980084 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a80b571_ce37_4ec8_ade7_80e670b49be2.slice/crio-1fc227c921b313676a9da35a8206f6105b825ba97e620c5e7e233bbe764fc7c3 WatchSource:0}: Error finding container 1fc227c921b313676a9da35a8206f6105b825ba97e620c5e7e233bbe764fc7c3: Status 404 returned error can't find the container with id 1fc227c921b313676a9da35a8206f6105b825ba97e620c5e7e233bbe764fc7c3 Oct 30 22:35:56.246180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:56.244229 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" event={"ID":"8a80b571-ce37-4ec8-ade7-80e670b49be2","Type":"ContainerStarted","Data":"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0"} Oct 30 22:35:56.246180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:56.244282 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" event={"ID":"8a80b571-ce37-4ec8-ade7-80e670b49be2","Type":"ContainerStarted","Data":"1fc227c921b313676a9da35a8206f6105b825ba97e620c5e7e233bbe764fc7c3"} Oct 30 22:35:56.367405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:56.367350 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" podStartSLOduration=1.367338806 podStartE2EDuration="1.367338806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:56.365565995 +0000 UTC m=+2484.334385702" watchObservedRunningTime="2026-10-30 22:35:56.367338806 +0000 UTC m=+2484.336158370" Oct 30 22:35:56.569138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:56.569108 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:35:56.639760 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:35:56.639722 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:35:57.252350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.252275 4292 generic.go:356] "Generic (PLEG): container finished" podID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerID="63562757b4f9e233a4d6bcb02a261f5298ba45d2c4420cf70e7ff0bcfc4b22ce" exitCode=0 Oct 30 22:35:57.252499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.252349 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerDied","Data":"63562757b4f9e233a4d6bcb02a261f5298ba45d2c4420cf70e7ff0bcfc4b22ce"} Oct 30 22:35:57.252499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.252409 4292 scope.go:122] "RemoveContainer" containerID="157724133e1d3f23b74adef15d9fccfcb5faee18e1d1f8b924a51d00144c1f5e" Oct 30 22:35:57.533295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.533264 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:35:57.686430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686396 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686450 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686476 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686564 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686636 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686673 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686712 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686734 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686743 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686759 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:57.686946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.686790 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\") pod \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\" (UID: \"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a\") " Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687077 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687134 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687156 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687422 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687444 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687460 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687474 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.687499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.687489 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.688809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.688790 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:57.688993 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.688971 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:57.689297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.689280 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "kube-api-access-wczqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:57.689297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.689284 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy" pod "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" (UID: "3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:57.789179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.789100 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.789179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.789135 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczqc\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-kube-api-access-wczqc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.789179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.789151 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:57.789179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:57.789165 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:58.260210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:58.260178 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" Oct 30 22:35:58.260210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:58.260199 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m" event={"ID":"3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a","Type":"ContainerDied","Data":"824a9e1a092113f2b05118479ec2b170fcda1c6db0f343b17b35a7af8d346e9b"} Oct 30 22:35:58.260419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:58.260256 4292 scope.go:122] "RemoveContainer" containerID="63562757b4f9e233a4d6bcb02a261f5298ba45d2c4420cf70e7ff0bcfc4b22ce" Oct 30 22:35:58.482665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:58.482590 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:35:58.532713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:35:58.532686 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-075cbb-openshift-default-655db97969-hk62m"] Oct 30 22:36:00.031227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031187 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs"] Oct 30 22:36:00.031767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031752 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.031831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031768 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.031831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031780 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.031831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031785 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.031928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031907 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.031928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.031917 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" containerName="istio-proxy" Oct 30 22:36:00.039889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.039868 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.095516 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.095484 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs"] Oct 30 22:36:00.142094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.142060 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:00.142347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.142328 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:36:00.142441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.142342 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:00.144734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.144714 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:00.147230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.147205 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a" path="/var/lib/kubelet/pods/3c29bf03-9fd9-4a32-8cb7-ae6dd3021b8a/volumes" Oct 30 22:36:00.216214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.216182 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdf7\" (UniqueName: \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.216339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.216296 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.216468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.216451 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.317287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.317259 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.317430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.317301 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdf7\" (UniqueName: \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.317430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.317400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.318288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.318264 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.319704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.319681 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.380313 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.380280 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdf7\" (UniqueName: \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\") pod \"maas-api-revoke-sub-777eca59-r6brs\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:00.654860 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:00.654792 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:01.033747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:01.033713 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs"] Oct 30 22:36:01.034611 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:01.034497 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00fe2c6_b237_4803_b744_cd291ca947c3.slice/crio-ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68 WatchSource:0}: Error finding container ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68: Status 404 returned error can't find the container with id ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68 Oct 30 22:36:01.280996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:01.280921 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" event={"ID":"d00fe2c6-b237-4803-b744-cd291ca947c3","Type":"ContainerStarted","Data":"ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68"} Oct 30 22:36:02.289100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:02.289021 4292 generic.go:356] "Generic (PLEG): container finished" podID="d00fe2c6-b237-4803-b744-cd291ca947c3" containerID="fb00451b279fb2eccf28c355b7d2506c5cec11c7474ba5d57c8e233de1be9853" exitCode=0 Oct 30 22:36:02.289486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:02.289093 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" event={"ID":"d00fe2c6-b237-4803-b744-cd291ca947c3","Type":"ContainerDied","Data":"fb00451b279fb2eccf28c355b7d2506c5cec11c7474ba5d57c8e233de1be9853"} Oct 30 22:36:02.464175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:02.464141 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:36:03.442740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.442712 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:03.559376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.559306 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\") pod \"d00fe2c6-b237-4803-b744-cd291ca947c3\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " Oct 30 22:36:03.559526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.559435 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\" (UniqueName: \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\") pod \"d00fe2c6-b237-4803-b744-cd291ca947c3\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " Oct 30 22:36:03.559526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.559487 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\") pod \"d00fe2c6-b237-4803-b744-cd291ca947c3\" (UID: \"d00fe2c6-b237-4803-b744-cd291ca947c3\") " Oct 30 22:36:03.559897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.559874 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca" pod "d00fe2c6-b237-4803-b744-cd291ca947c3" (UID: "d00fe2c6-b237-4803-b744-cd291ca947c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:03.559985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.559965 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:03.561990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.561967 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca" pod "d00fe2c6-b237-4803-b744-cd291ca947c3" (UID: "d00fe2c6-b237-4803-b744-cd291ca947c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:03.564123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.564097 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7" pod "d00fe2c6-b237-4803-b744-cd291ca947c3" (UID: "d00fe2c6-b237-4803-b744-cd291ca947c3"). InnerVolumeSpecName "kube-api-access-fzdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:03.660871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.660840 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00fe2c6-b237-4803-b744-cd291ca947c3-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:03.660871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:03.660869 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzdf7\" (UniqueName: \"kubernetes.io/projected/d00fe2c6-b237-4803-b744-cd291ca947c3-kube-api-access-fzdf7\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:04.303151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:04.303121 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" Oct 30 22:36:04.303351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:04.303156 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs" event={"ID":"d00fe2c6-b237-4803-b744-cd291ca947c3","Type":"ContainerDied","Data":"ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68"} Oct 30 22:36:04.303351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:04.303204 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8e746f5d88830b6a77293cbebc55f2161d8ac0272c74cd6cfc496b9ca32d68" Oct 30 22:36:05.569197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:05.569157 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:36:05.581833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:05.581805 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:36:06.325906 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:06.325873 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:36:07.464978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:07.464940 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" containerID="cri-o://bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924" gracePeriod=25 Oct 30 22:36:10.275662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:10.275617 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:10.884998 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:10.884957 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" containerID="cri-o://00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049" gracePeriod=45 Oct 30 22:36:11.151988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.151923 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:36:11.247282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247290 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247381 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247467 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\" (UniqueName: \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247519 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247567 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\") pod \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\" (UID: \"7508f48e-dc12-4184-8fe8-a243ad9fd4aa\") " Oct 30 22:36:11.247693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247649 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.247826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247800 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.247826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.247812 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.248106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.248084 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.248149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.248113 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.248149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.248126 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.249604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.249554 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "kube-api-access-cqzjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:11.249705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.249615 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:11.249705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.249628 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm" pod "7508f48e-dc12-4184-8fe8-a243ad9fd4aa" (UID: "7508f48e-dc12-4184-8fe8-a243ad9fd4aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.320474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.320440 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf"] Oct 30 22:36:11.321217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321194 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d00fe2c6-b237-4803-b744-cd291ca947c3" containerName="revoke-keys" Oct 30 22:36:11.321217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321218 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00fe2c6-b237-4803-b744-cd291ca947c3" containerName="revoke-keys" Oct 30 22:36:11.321423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321271 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" Oct 30 22:36:11.321423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321279 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" Oct 30 22:36:11.321423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321417 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerName="main" Oct 30 22:36:11.321617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.321439 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00fe2c6-b237-4803-b744-cd291ca947c3" containerName="revoke-keys" Oct 30 22:36:11.325297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.325276 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.349816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.349786 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzjn\" (UniqueName: \"kubernetes.io/projected/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-kube-api-access-cqzjn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.349971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.349819 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.349971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.349855 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7508f48e-dc12-4184-8fe8-a243ad9fd4aa-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.360093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.360060 4292 generic.go:356] "Generic (PLEG): container finished" podID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" containerID="00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049" exitCode=0 Oct 30 22:36:11.360222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.360105 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" Oct 30 22:36:11.360222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.360142 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" event={"ID":"7508f48e-dc12-4184-8fe8-a243ad9fd4aa","Type":"ContainerDied","Data":"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049"} Oct 30 22:36:11.360222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.360193 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99" event={"ID":"7508f48e-dc12-4184-8fe8-a243ad9fd4aa","Type":"ContainerDied","Data":"7be635b66508e681489861fcabf157ce1a17bcfc66cdc8e88a833f2ee906b928"} Oct 30 22:36:11.360222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.360215 4292 scope.go:122] "RemoveContainer" containerID="00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049" Oct 30 22:36:11.372145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.372127 4292 scope.go:122] "RemoveContainer" containerID="00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049" Oct 30 22:36:11.372424 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:11.372404 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049\": container with ID starting with 00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049 not found: ID does not exist" containerID="00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049" Oct 30 22:36:11.372494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.372434 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049"} err="failed to get container status \"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049\": rpc error: code = NotFound desc = could not find container \"00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049\": container with ID starting with 00f400b7ef88356e5da6749a01adb6061ed895b03652aa3ec2a5fdc3905be049 not found: ID does not exist" Oct 30 22:36:11.380195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.380175 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:11.380488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.380464 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:11.380543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.380472 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:11.380739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.380723 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:36:11.390401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.390379 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf"] Oct 30 22:36:11.451322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.451241 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.451465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.451381 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.451524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.451481 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqrt\" (UniqueName: \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.553083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.553007 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqrt\" (UniqueName: \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.553328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.553131 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.553328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.553253 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.554790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.554433 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.557077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.556819 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.560068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.560045 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:36:11.567361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.567338 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2a4123/distinct-w0-2a4123-kserve-9cd579688-8dr99"] Oct 30 22:36:11.616622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.616593 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqrt\" (UniqueName: \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\") pod \"maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:11.643167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:11.643146 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:12.024801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:12.024767 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf"] Oct 30 22:36:12.025030 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:12.025008 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e83241_bf6b_49fc_97db_26beacc87595.slice/crio-154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284 WatchSource:0}: Error finding container 154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284: Status 404 returned error can't find the container with id 154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284 Oct 30 22:36:12.147568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:12.147542 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7508f48e-dc12-4184-8fe8-a243ad9fd4aa" path="/var/lib/kubelet/pods/7508f48e-dc12-4184-8fe8-a243ad9fd4aa/volumes" Oct 30 22:36:12.375642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:12.375597 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" event={"ID":"c4e83241-bf6b-49fc-97db-26beacc87595","Type":"ContainerStarted","Data":"888d23cd81d89d473216082b938e923e0bb081fbdaf6759c00ec334e949d24e9"} Oct 30 22:36:12.376021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:12.375696 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" event={"ID":"c4e83241-bf6b-49fc-97db-26beacc87595","Type":"ContainerStarted","Data":"154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284"} Oct 30 22:36:12.499815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:12.499772 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" podStartSLOduration=1.499759351 podStartE2EDuration="1.499759351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:12.497618796 +0000 UTC m=+2500.466438358" watchObservedRunningTime="2026-10-30 22:36:12.499759351 +0000 UTC m=+2500.468578914" Oct 30 22:36:13.384144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:13.384109 4292 generic.go:356] "Generic (PLEG): container finished" podID="c4e83241-bf6b-49fc-97db-26beacc87595" containerID="888d23cd81d89d473216082b938e923e0bb081fbdaf6759c00ec334e949d24e9" exitCode=0 Oct 30 22:36:13.384675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:13.384155 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" event={"ID":"c4e83241-bf6b-49fc-97db-26beacc87595","Type":"ContainerDied","Data":"888d23cd81d89d473216082b938e923e0bb081fbdaf6759c00ec334e949d24e9"} Oct 30 22:36:14.527262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.527236 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:14.591213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.591181 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\") pod \"c4e83241-bf6b-49fc-97db-26beacc87595\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " Oct 30 22:36:14.591364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.591223 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\" (UniqueName: \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\") pod \"c4e83241-bf6b-49fc-97db-26beacc87595\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " Oct 30 22:36:14.591364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.591334 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\") pod \"c4e83241-bf6b-49fc-97db-26beacc87595\" (UID: \"c4e83241-bf6b-49fc-97db-26beacc87595\") " Oct 30 22:36:14.591744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.591721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca" pod "c4e83241-bf6b-49fc-97db-26beacc87595" (UID: "c4e83241-bf6b-49fc-97db-26beacc87595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:14.591916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.591900 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:14.593417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.593398 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca" pod "c4e83241-bf6b-49fc-97db-26beacc87595" (UID: "c4e83241-bf6b-49fc-97db-26beacc87595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:14.593511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.593441 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt" pod "c4e83241-bf6b-49fc-97db-26beacc87595" (UID: "c4e83241-bf6b-49fc-97db-26beacc87595"). InnerVolumeSpecName "kube-api-access-mnqrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:14.692566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.692503 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e83241-bf6b-49fc-97db-26beacc87595-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:14.692566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:14.692530 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqrt\" (UniqueName: \"kubernetes.io/projected/c4e83241-bf6b-49fc-97db-26beacc87595-kube-api-access-mnqrt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:15.398160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:15.398128 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" Oct 30 22:36:15.398315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:15.398164 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf" event={"ID":"c4e83241-bf6b-49fc-97db-26beacc87595","Type":"ContainerDied","Data":"154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284"} Oct 30 22:36:15.398315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:15.398219 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154c938d96518a5f2f9d06c0cd2a7ce071d9e6837e6e741ef4ad717578bed284" Oct 30 22:36:15.802152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:15.802114 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:36:20.295599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:20.293663 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:22.926114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:22.926076 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:36:26.639723 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:26.639688 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:36:27.725707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.725639 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:36:27.795878 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.795847 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\") pod \"30d8fe97-233f-4841-9d5b-51255fc08384\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " Oct 30 22:36:27.796068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.795913 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\" (UniqueName: \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\") pod \"30d8fe97-233f-4841-9d5b-51255fc08384\" (UID: \"30d8fe97-233f-4841-9d5b-51255fc08384\") " Oct 30 22:36:27.796311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.796285 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume" pod "30d8fe97-233f-4841-9d5b-51255fc08384" (UID: "30d8fe97-233f-4841-9d5b-51255fc08384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:27.798385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.798363 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth" pod "30d8fe97-233f-4841-9d5b-51255fc08384" (UID: "30d8fe97-233f-4841-9d5b-51255fc08384"). InnerVolumeSpecName "kube-api-access-crsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:27.897083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.897050 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d8fe97-233f-4841-9d5b-51255fc08384-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:27.897083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.897082 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crsth\" (UniqueName: \"kubernetes.io/projected/30d8fe97-233f-4841-9d5b-51255fc08384-kube-api-access-crsth\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:27.926792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:27.926738 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" podUID="40c8267c-cca3-4e4d-8b41-4291130d9e97" containerName="payload-processing" containerID="cri-o://c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e" gracePeriod=25 Oct 30 22:36:28.170742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.170717 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:36:28.199931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.199893 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\") pod \"40c8267c-cca3-4e4d-8b41-4291130d9e97\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " Oct 30 22:36:28.200062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.199948 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\" (UniqueName: \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\") pod \"40c8267c-cca3-4e4d-8b41-4291130d9e97\" (UID: \"40c8267c-cca3-4e4d-8b41-4291130d9e97\") " Oct 30 22:36:28.200241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.200220 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume" pod "40c8267c-cca3-4e4d-8b41-4291130d9e97" (UID: "40c8267c-cca3-4e4d-8b41-4291130d9e97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:28.200344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.200332 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c8267c-cca3-4e4d-8b41-4291130d9e97-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:28.201928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.201910 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl" pod "40c8267c-cca3-4e4d-8b41-4291130d9e97" (UID: "40c8267c-cca3-4e4d-8b41-4291130d9e97"). InnerVolumeSpecName "kube-api-access-kvtxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:28.301848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.301815 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtxl\" (UniqueName: \"kubernetes.io/projected/40c8267c-cca3-4e4d-8b41-4291130d9e97-kube-api-access-kvtxl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:28.526960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.526924 4292 generic.go:356] "Generic (PLEG): container finished" podID="40c8267c-cca3-4e4d-8b41-4291130d9e97" containerID="c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e" exitCode=0 Oct 30 22:36:28.527117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.526976 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" Oct 30 22:36:28.527117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.527013 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" event={"ID":"40c8267c-cca3-4e4d-8b41-4291130d9e97","Type":"ContainerDied","Data":"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e"} Oct 30 22:36:28.527117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.527067 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq" event={"ID":"40c8267c-cca3-4e4d-8b41-4291130d9e97","Type":"ContainerDied","Data":"1ee456924cde29710f565a27818bc2e2dedec4401ced1366766f77a1b92a81e7"} Oct 30 22:36:28.527117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.527096 4292 scope.go:122] "RemoveContainer" containerID="c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e" Oct 30 22:36:28.528811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.528785 4292 generic.go:356] "Generic (PLEG): container finished" podID="30d8fe97-233f-4841-9d5b-51255fc08384" containerID="bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924" exitCode=0 Oct 30 22:36:28.528952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.528829 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" Oct 30 22:36:28.528952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.528834 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" event={"ID":"30d8fe97-233f-4841-9d5b-51255fc08384","Type":"ContainerDied","Data":"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924"} Oct 30 22:36:28.528952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.528874 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9" event={"ID":"30d8fe97-233f-4841-9d5b-51255fc08384","Type":"ContainerDied","Data":"3ad61de9160131ae3f397d7f73ce77fb85617ea382797d5c637b1ede65282479"} Oct 30 22:36:28.537919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.537901 4292 scope.go:122] "RemoveContainer" containerID="c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e" Oct 30 22:36:28.538210 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:28.538189 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e\": container with ID starting with c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e not found: ID does not exist" containerID="c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e" Oct 30 22:36:28.538264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.538217 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e"} err="failed to get container status \"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e\": rpc error: code = NotFound desc = could not find container \"c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e\": container with ID starting with c35bbebd0de86564c447901eaa0d5087e6f97b0d34da1461e6c4ceef5c13863e not found: ID does not exist" Oct 30 22:36:28.538264 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.538236 4292 scope.go:122] "RemoveContainer" containerID="bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924" Oct 30 22:36:28.546879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.546858 4292 scope.go:122] "RemoveContainer" containerID="bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924" Oct 30 22:36:28.547145 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:28.547123 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924\": container with ID starting with bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924 not found: ID does not exist" containerID="bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924" Oct 30 22:36:28.547201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.547152 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924"} err="failed to get container status \"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924\": rpc error: code = NotFound desc = could not find container \"bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924\": container with ID starting with bf5e4bb915f4d268569001d1170efb2b824e12b95818c30f0b155eec9c030924 not found: ID does not exist" Oct 30 22:36:28.698503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.698430 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:36:28.704400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.704370 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-97aa64-59b6b9d7c5-8r8cq"] Oct 30 22:36:28.819333 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.819300 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:36:28.825088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:28.825064 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2a4123-85dc96866c-ck9f9"] Oct 30 22:36:30.105796 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.105766 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl"] Oct 30 22:36:30.106280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106267 4292 cpu_manager.go:414] "containerMap: removing container" podUID="40c8267c-cca3-4e4d-8b41-4291130d9e97" containerName="payload-processing" Oct 30 22:36:30.106321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106281 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c8267c-cca3-4e4d-8b41-4291130d9e97" containerName="payload-processing" Oct 30 22:36:30.106321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106311 4292 cpu_manager.go:414] "containerMap: removing container" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" Oct 30 22:36:30.106321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106318 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" Oct 30 22:36:30.106425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106336 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c4e83241-bf6b-49fc-97db-26beacc87595" containerName="revoke-keys" Oct 30 22:36:30.106425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106341 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e83241-bf6b-49fc-97db-26beacc87595" containerName="revoke-keys" Oct 30 22:36:30.106425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106409 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c8267c-cca3-4e4d-8b41-4291130d9e97" containerName="payload-processing" Oct 30 22:36:30.106425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106422 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" containerName="payload-pre-processing" Oct 30 22:36:30.106556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.106430 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e83241-bf6b-49fc-97db-26beacc87595" containerName="revoke-keys" Oct 30 22:36:30.110994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.110974 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.146144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.146117 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d8fe97-233f-4841-9d5b-51255fc08384" path="/var/lib/kubelet/pods/30d8fe97-233f-4841-9d5b-51255fc08384/volumes" Oct 30 22:36:30.146504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.146491 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c8267c-cca3-4e4d-8b41-4291130d9e97" path="/var/lib/kubelet/pods/40c8267c-cca3-4e4d-8b41-4291130d9e97/volumes" Oct 30 22:36:30.162168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.162143 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx"] Oct 30 22:36:30.166190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.166173 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.167892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.167871 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl"] Oct 30 22:36:30.210210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.209901 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:30.210210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.209929 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:30.210210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.209958 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:30.215539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.215520 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:36:30.220062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.220036 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qx9\" (UniqueName: \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.220173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.220066 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.221076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.221047 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.221198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.221119 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjct\" (UniqueName: \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.221198 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.221183 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.221306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.221255 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.225176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.225152 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx"] Oct 30 22:36:30.322880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.322845 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.323056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.322892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjct\" (UniqueName: \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.323056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.322921 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.323056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.322941 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.323056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.323001 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qx9\" (UniqueName: \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.323056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.323026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.323694 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.323669 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.323801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.323780 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.325407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.325379 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.325510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.325407 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.386242 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.386181 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qx9\" (UniqueName: \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\") pod \"maas-api-revoke-sub-8f6b24e1-5n8wl\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.386360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.386338 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjct\" (UniqueName: \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\") pod \"maas-api-revoke-sub-ccdbfac0-jrxtx\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.428110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.428086 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:30.479225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.479186 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:30.812094 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:30.812062 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680e2501_67fe_4568_bdf8_041dc19b410c.slice/crio-aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690 WatchSource:0}: Error finding container aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690: Status 404 returned error can't find the container with id aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690 Oct 30 22:36:30.812199 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.812135 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl"] Oct 30 22:36:30.822613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.822566 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" containerID="cri-o://9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751" gracePeriod=45 Oct 30 22:36:30.870006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:30.869983 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx"] Oct 30 22:36:30.895822 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:30.895794 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83569574_1464_4a63_8c70_8b328e5be71f.slice/crio-a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410 WatchSource:0}: Error finding container a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410: Status 404 returned error can't find the container with id a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410 Oct 30 22:36:31.071305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.071281 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:36:31.136665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136636 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\" (UniqueName: \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136674 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136814 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136909 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.137051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136914 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\" (UniqueName: \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.136968 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137057 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.137292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137108 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\") pod \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\" (UID: \"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce\") " Oct 30 22:36:31.137379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137358 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.137559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137544 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.137650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137562 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.137650 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.137589 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.185142 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.185115 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "kube-api-access-z8wjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.185258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.185179 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.185258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.185188 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs" pod "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" (UID: "98e43ef4-8e55-4eb6-ab48-41b6d66f9fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:31.239366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.239330 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.239517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.239372 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wjr\" (UniqueName: \"kubernetes.io/projected/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-kube-api-access-z8wjr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.239517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.239389 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.556450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.556409 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" event={"ID":"83569574-1464-4a63-8c70-8b328e5be71f","Type":"ContainerStarted","Data":"f7ad75fbc8c8f36d80ee45a0329b014eb93d37c743e82ff6ecb7ad7017e60f8c"} Oct 30 22:36:31.556667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.556466 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" event={"ID":"83569574-1464-4a63-8c70-8b328e5be71f","Type":"ContainerStarted","Data":"a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410"} Oct 30 22:36:31.557831 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.557806 4292 generic.go:356] "Generic (PLEG): container finished" podID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerID="9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751" exitCode=0 Oct 30 22:36:31.557967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.557839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" Oct 30 22:36:31.557967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.557892 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" event={"ID":"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce","Type":"ContainerDied","Data":"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751"} Oct 30 22:36:31.557967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.557936 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm" event={"ID":"98e43ef4-8e55-4eb6-ab48-41b6d66f9fce","Type":"ContainerDied","Data":"6782f961a978b196d915e2f57b661ad1ab158f0c474421950877dc02735cd2fa"} Oct 30 22:36:31.557967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.557958 4292 scope.go:122] "RemoveContainer" containerID="9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751" Oct 30 22:36:31.559760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.559738 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" event={"ID":"680e2501-67fe-4568-bdf8-041dc19b410c","Type":"ContainerStarted","Data":"cdad92dace2347b70db0cf77006313b38cf9bb2c2fa7bbc2de03778a83aeb444"} Oct 30 22:36:31.559875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.559789 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" event={"ID":"680e2501-67fe-4568-bdf8-041dc19b410c","Type":"ContainerStarted","Data":"aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690"} Oct 30 22:36:31.568778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.568757 4292 scope.go:122] "RemoveContainer" containerID="9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751" Oct 30 22:36:31.569034 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:31.569004 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751\": container with ID starting with 9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751 not found: ID does not exist" containerID="9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751" Oct 30 22:36:31.569126 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.569040 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751"} err="failed to get container status \"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751\": rpc error: code = NotFound desc = could not find container \"9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751\": container with ID starting with 9f54845604069435524e9b1087ce6aa20d1baa9ac5fb14e39e7926009ad3d751 not found: ID does not exist" Oct 30 22:36:31.677928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.677831 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" podStartSLOduration=1.677816168 podStartE2EDuration="1.677816168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:31.676431041 +0000 UTC m=+2519.645250606" watchObservedRunningTime="2026-10-30 22:36:31.677816168 +0000 UTC m=+2519.646635731" Oct 30 22:36:31.793522 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.793494 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:36:31.801070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.801044 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-97aa64/simulator-w4-97aa64-kserve-7d57d94fbc-795tm"] Oct 30 22:36:31.922062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:31.922008 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" podStartSLOduration=1.921994331 podStartE2EDuration="1.921994331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:31.919918898 +0000 UTC m=+2519.888738461" watchObservedRunningTime="2026-10-30 22:36:31.921994331 +0000 UTC m=+2519.890813895" Oct 30 22:36:32.147880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:32.147849 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" path="/var/lib/kubelet/pods/98e43ef4-8e55-4eb6-ab48-41b6d66f9fce/volumes" Oct 30 22:36:32.567480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:32.567441 4292 generic.go:356] "Generic (PLEG): container finished" podID="83569574-1464-4a63-8c70-8b328e5be71f" containerID="f7ad75fbc8c8f36d80ee45a0329b014eb93d37c743e82ff6ecb7ad7017e60f8c" exitCode=0 Oct 30 22:36:32.567671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:32.567526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" event={"ID":"83569574-1464-4a63-8c70-8b328e5be71f","Type":"ContainerDied","Data":"f7ad75fbc8c8f36d80ee45a0329b014eb93d37c743e82ff6ecb7ad7017e60f8c"} Oct 30 22:36:32.570181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:32.570154 4292 generic.go:356] "Generic (PLEG): container finished" podID="680e2501-67fe-4568-bdf8-041dc19b410c" containerID="cdad92dace2347b70db0cf77006313b38cf9bb2c2fa7bbc2de03778a83aeb444" exitCode=0 Oct 30 22:36:32.570304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:32.570274 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" event={"ID":"680e2501-67fe-4568-bdf8-041dc19b410c","Type":"ContainerDied","Data":"cdad92dace2347b70db0cf77006313b38cf9bb2c2fa7bbc2de03778a83aeb444"} Oct 30 22:36:33.758938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.758909 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:33.761136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.761118 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:33.867108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867017 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\" (UniqueName: \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\") pod \"83569574-1464-4a63-8c70-8b328e5be71f\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " Oct 30 22:36:33.867312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867107 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\") pod \"83569574-1464-4a63-8c70-8b328e5be71f\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " Oct 30 22:36:33.867312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867162 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\" (UniqueName: \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\") pod \"680e2501-67fe-4568-bdf8-041dc19b410c\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " Oct 30 22:36:33.867312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867233 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\") pod \"680e2501-67fe-4568-bdf8-041dc19b410c\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " Oct 30 22:36:33.867510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867337 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\") pod \"680e2501-67fe-4568-bdf8-041dc19b410c\" (UID: \"680e2501-67fe-4568-bdf8-041dc19b410c\") " Oct 30 22:36:33.867510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867448 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\") pod \"83569574-1464-4a63-8c70-8b328e5be71f\" (UID: \"83569574-1464-4a63-8c70-8b328e5be71f\") " Oct 30 22:36:33.867648 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867489 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca" pod "83569574-1464-4a63-8c70-8b328e5be71f" (UID: "83569574-1464-4a63-8c70-8b328e5be71f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:33.867728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.867703 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca" pod "680e2501-67fe-4568-bdf8-041dc19b410c" (UID: "680e2501-67fe-4568-bdf8-041dc19b410c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:33.868038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.868018 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:33.868134 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.868043 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:33.869455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.869427 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9" pod "680e2501-67fe-4568-bdf8-041dc19b410c" (UID: "680e2501-67fe-4568-bdf8-041dc19b410c"). InnerVolumeSpecName "kube-api-access-68qx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:33.869622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.869590 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca" pod "83569574-1464-4a63-8c70-8b328e5be71f" (UID: "83569574-1464-4a63-8c70-8b328e5be71f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:33.869981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.869951 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca" pod "680e2501-67fe-4568-bdf8-041dc19b410c" (UID: "680e2501-67fe-4568-bdf8-041dc19b410c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:33.869981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.869968 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct" pod "83569574-1464-4a63-8c70-8b328e5be71f" (UID: "83569574-1464-4a63-8c70-8b328e5be71f"). InnerVolumeSpecName "kube-api-access-mzjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:33.968763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.968735 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qx9\" (UniqueName: \"kubernetes.io/projected/680e2501-67fe-4568-bdf8-041dc19b410c-kube-api-access-68qx9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:33.968763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.968761 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680e2501-67fe-4568-bdf8-041dc19b410c-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:33.968908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.968770 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83569574-1464-4a63-8c70-8b328e5be71f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:33.968908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:33.968780 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjct\" (UniqueName: \"kubernetes.io/projected/83569574-1464-4a63-8c70-8b328e5be71f-kube-api-access-mzjct\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:34.584966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.584939 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" Oct 30 22:36:34.585138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.584970 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx" event={"ID":"83569574-1464-4a63-8c70-8b328e5be71f","Type":"ContainerDied","Data":"a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410"} Oct 30 22:36:34.585138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.585039 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75e0f9bc18d9dc10bacad6cb5a1b21dae0090640e32a01193f43ff6131b1410" Oct 30 22:36:34.586628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.586607 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" Oct 30 22:36:34.586755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.586605 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl" event={"ID":"680e2501-67fe-4568-bdf8-041dc19b410c","Type":"ContainerDied","Data":"aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690"} Oct 30 22:36:34.586755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:34.586726 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf4787c782068658f0d01306ddf783ddd10d5b8fb6942dd75c0ac448924d690" Oct 30 22:36:37.508928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.508896 4292 scope.go:122] "RemoveContainer" containerID="c1174c6d939b393e8673558f66ed7db2b46724aea4d4d90dc8faaa83fc840907" Oct 30 22:36:37.518503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.518481 4292 scope.go:122] "RemoveContainer" containerID="90fd57258844811a8f73aa4a1c1acb7e2a164ba9d896244958e05c6e71225b04" Oct 30 22:36:37.529704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.529684 4292 scope.go:122] "RemoveContainer" containerID="d41faffa138d359c21271687cec3c89de20ae8bb6fd50195a351862673e9a3cd" Oct 30 22:36:37.538072 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.538046 4292 scope.go:122] "RemoveContainer" containerID="85403ddd395fb766f8a96f793f068067157ca56ccb54ceb2c66e8b4689d4a476" Oct 30 22:36:37.548605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.548584 4292 scope.go:122] "RemoveContainer" containerID="fe8ff60fbc8dcf156981639536dbc95804110e512826f525a74ed137a317675f" Oct 30 22:36:37.556763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:37.556746 4292 scope.go:122] "RemoveContainer" containerID="35d513db3787dbb5d3f2b04e2420db17084f930a091c98a52ff2a2d767417f70" Oct 30 22:36:45.638204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638171 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx"] Oct 30 22:36:45.638892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638872 4292 cpu_manager.go:414] "containerMap: removing container" podUID="680e2501-67fe-4568-bdf8-041dc19b410c" containerName="revoke-keys" Oct 30 22:36:45.638996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638894 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e2501-67fe-4568-bdf8-041dc19b410c" containerName="revoke-keys" Oct 30 22:36:45.638996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638920 4292 cpu_manager.go:414] "containerMap: removing container" podUID="83569574-1464-4a63-8c70-8b328e5be71f" containerName="revoke-keys" Oct 30 22:36:45.638996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638930 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83569574-1464-4a63-8c70-8b328e5be71f" containerName="revoke-keys" Oct 30 22:36:45.638996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.638995 4292 cpu_manager.go:414] "containerMap: removing container" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" Oct 30 22:36:45.639206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.639003 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" Oct 30 22:36:45.639206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.639143 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e2501-67fe-4568-bdf8-041dc19b410c" containerName="revoke-keys" Oct 30 22:36:45.639206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.639160 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="83569574-1464-4a63-8c70-8b328e5be71f" containerName="revoke-keys" Oct 30 22:36:45.639206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.639179 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e43ef4-8e55-4eb6-ab48-41b6d66f9fce" containerName="main" Oct 30 22:36:45.644646 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.644623 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.700299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.700271 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445"] Oct 30 22:36:45.704053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.704037 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.747704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.747678 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:45.750500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.750463 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:45.750500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.750461 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:45.750731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.750500 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:36:45.750989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.750967 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx"] Oct 30 22:36:45.766145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.766082 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445"] Oct 30 22:36:45.814729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.814701 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.814869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.814743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.814869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.814777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsrj\" (UniqueName: \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.814968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.814918 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.815012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.814982 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88kns\" (UniqueName: \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.815048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.815010 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.916459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916372 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.916459 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916424 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.916739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916533 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsrj\" (UniqueName: \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.916739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916653 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.916739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916704 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88kns\" (UniqueName: \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.916944 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.916798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.917464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.917436 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.917623 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.917470 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.918825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.918805 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:45.918907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.918845 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.980096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.980069 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88kns\" (UniqueName: \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\") pod \"maas-api-revoke-sub-beea9ec0-7xkzx\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:45.980241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:45.980200 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsrj\" (UniqueName: \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\") pod \"maas-api-revoke-sub-438de563-dh445\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:46.017798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.017774 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:46.258566 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.258480 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:46.404540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.404511 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445"] Oct 30 22:36:46.405238 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:46.405211 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa64f4f_4fdb_4d22_a0d9_ac1b57a451c0.slice/crio-c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8 WatchSource:0}: Error finding container c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8: Status 404 returned error can't find the container with id c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8 Oct 30 22:36:46.636861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.636837 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx"] Oct 30 22:36:46.637278 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:36:46.637252 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dd4e74_a9f0_456a_9fce_b0b207d6bc93.slice/crio-22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f WatchSource:0}: Error finding container 22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f: Status 404 returned error can't find the container with id 22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f Oct 30 22:36:46.664060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.664033 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" event={"ID":"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93","Type":"ContainerStarted","Data":"22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f"} Oct 30 22:36:46.665254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:46.665222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" event={"ID":"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0","Type":"ContainerStarted","Data":"c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8"} Oct 30 22:36:47.673598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:47.673499 4292 generic.go:356] "Generic (PLEG): container finished" podID="e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" containerID="8c096a38f271c3d0cb3f9071c05f84fcff9dc0b371f6f2dbcf4f94e67bad2073" exitCode=0 Oct 30 22:36:47.674033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:47.673605 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" event={"ID":"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93","Type":"ContainerDied","Data":"8c096a38f271c3d0cb3f9071c05f84fcff9dc0b371f6f2dbcf4f94e67bad2073"} Oct 30 22:36:47.675418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:47.675388 4292 generic.go:356] "Generic (PLEG): container finished" podID="9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" containerID="cbcd267d6f6ad3206ded1e5efda702c4ce4707243107d22bde32c1a0a13f2475" exitCode=0 Oct 30 22:36:47.675601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:47.675467 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" event={"ID":"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0","Type":"ContainerDied","Data":"cbcd267d6f6ad3206ded1e5efda702c4ce4707243107d22bde32c1a0a13f2475"} Oct 30 22:36:48.829557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.829209 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:48.867867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.867844 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:48.950469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950406 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\") pod \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " Oct 30 22:36:48.950637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950488 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\" (UniqueName: \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\") pod \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " Oct 30 22:36:48.950637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950565 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\") pod \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " Oct 30 22:36:48.950770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950671 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\" (UniqueName: \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\") pod \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " Oct 30 22:36:48.950770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950732 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\") pod \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\" (UID: \"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0\") " Oct 30 22:36:48.950870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.950774 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\") pod \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\" (UID: \"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93\") " Oct 30 22:36:48.951410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.951331 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca" pod "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" (UID: "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:48.951543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.951518 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca" pod "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" (UID: "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:48.953274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.953253 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca" pod "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" (UID: "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:48.953379 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.953330 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj" pod "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" (UID: "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0"). InnerVolumeSpecName "kube-api-access-pcsrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:48.953501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.953482 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns" pod "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" (UID: "e9dd4e74-a9f0-456a-9fce-b0b207d6bc93"). InnerVolumeSpecName "kube-api-access-88kns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:48.954347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:48.954309 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca" pod "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" (UID: "9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:49.052345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052317 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.052345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052344 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.052536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052355 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.052536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052364 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcsrj\" (UniqueName: \"kubernetes.io/projected/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-kube-api-access-pcsrj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.052536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052373 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.052536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.052382 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88kns\" (UniqueName: \"kubernetes.io/projected/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93-kube-api-access-88kns\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.688343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.688300 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" event={"ID":"e9dd4e74-a9f0-456a-9fce-b0b207d6bc93","Type":"ContainerDied","Data":"22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f"} Oct 30 22:36:49.688536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.688356 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f8105f3e0e74ba49595feb2216edc7e9d73db59528854e655cd47818152e8f" Oct 30 22:36:49.688536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.688310 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx" Oct 30 22:36:49.689735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.689712 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" Oct 30 22:36:49.689861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.689747 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445" event={"ID":"9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0","Type":"ContainerDied","Data":"c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8"} Oct 30 22:36:49.689861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:36:49.689789 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c294a1c2a4849a86218972dcc17d245429fd40e860dd8dad658b4dd52c42cee8" Oct 30 22:36:56.640515 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:36:56.640476 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:37:00.067311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.067263 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6"] Oct 30 22:37:00.113638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.113600 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl"] Oct 30 22:37:00.116431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.116404 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4f6954-zkhbl"] Oct 30 22:37:00.119415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.119389 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69069d29-drxd6"] Oct 30 22:37:00.146103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.146073 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0bfc7d-7252-4559-a66e-38d50da85f68" path="/var/lib/kubelet/pods/7a0bfc7d-7252-4559-a66e-38d50da85f68/volumes" Oct 30 22:37:00.146530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:00.146516 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d" path="/var/lib/kubelet/pods/c50a9868-ed11-4ee4-9ce6-ad98ea2e6a5d/volumes" Oct 30 22:37:08.418514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:08.418478 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk"] Oct 30 22:37:08.469358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:08.469323 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2a4123-dzljk"] Oct 30 22:37:10.146274 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:10.146245 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4306e3ba-778d-4c3e-a10a-b75784768391" path="/var/lib/kubelet/pods/4306e3ba-778d-4c3e-a10a-b75784768391/volumes" Oct 30 22:37:16.451383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:16.451299 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:37:16.451869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:16.451563 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" containerID="cri-o://f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837" gracePeriod=30 Oct 30 22:37:20.030896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:20.030851 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:37:20.031409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:20.030938 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:37:21.711608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.711560 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:37:21.812298 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812273 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812318 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812345 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812386 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812475 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812563 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812605 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812627 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812679 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812706 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:21.812748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812731 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\") pod \"86002a59-1b82-4516-8b86-c3251d6936cb\" (UID: \"86002a59-1b82-4516-8b86-c3251d6936cb\") " Oct 30 22:37:21.813081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812850 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:21.813081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.812872 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:21.813081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813066 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:21.813324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813297 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.813324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813318 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.813428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813332 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86002a59-1b82-4516-8b86-c3251d6936cb-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.813428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813345 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.813428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.813360 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.814793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.814769 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:21.815009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.814980 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:21.815121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.815029 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:21.815215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.815191 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54" pod "86002a59-1b82-4516-8b86-c3251d6936cb" (UID: "86002a59-1b82-4516-8b86-c3251d6936cb"). InnerVolumeSpecName "kube-api-access-nnz54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:21.893354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.893317 4292 generic.go:356] "Generic (PLEG): container finished" podID="86002a59-1b82-4516-8b86-c3251d6936cb" containerID="f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837" exitCode=0 Oct 30 22:37:21.893515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.893370 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" Oct 30 22:37:21.893515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.893406 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerDied","Data":"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837"} Oct 30 22:37:21.893515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.893459 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck" event={"ID":"86002a59-1b82-4516-8b86-c3251d6936cb","Type":"ContainerDied","Data":"0cc239777507eb4315c5ce632cdc0dda12ab903ac9a803cde075fcaf560f2205"} Oct 30 22:37:21.893515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.893481 4292 scope.go:122] "RemoveContainer" containerID="f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837" Oct 30 22:37:21.903962 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.903944 4292 scope.go:122] "RemoveContainer" containerID="badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158" Oct 30 22:37:21.912157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.912134 4292 scope.go:122] "RemoveContainer" containerID="f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837" Oct 30 22:37:21.912400 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:37:21.912381 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837\": container with ID starting with f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837 not found: ID does not exist" containerID="f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837" Oct 30 22:37:21.912457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.912407 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837"} err="failed to get container status \"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837\": rpc error: code = NotFound desc = could not find container \"f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837\": container with ID starting with f83ecc1a3efeb0209db79ef0abbe0d3fce2877e04f15bd4826f92348c9d32837 not found: ID does not exist" Oct 30 22:37:21.912457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.912423 4292 scope.go:122] "RemoveContainer" containerID="badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158" Oct 30 22:37:21.912676 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:37:21.912658 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158\": container with ID starting with badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158 not found: ID does not exist" containerID="badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158" Oct 30 22:37:21.912726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.912681 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158"} err="failed to get container status \"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158\": rpc error: code = NotFound desc = could not find container \"badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158\": container with ID starting with badaadd1d63051d7f483cfe69db009d74083c567dc5bbf3085ffcbe0f3456158 not found: ID does not exist" Oct 30 22:37:21.914336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.914312 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86002a59-1b82-4516-8b86-c3251d6936cb-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.914428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.914337 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.914428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.914352 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86002a59-1b82-4516-8b86-c3251d6936cb-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.914428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:21.914365 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnz54\" (UniqueName: \"kubernetes.io/projected/86002a59-1b82-4516-8b86-c3251d6936cb-kube-api-access-nnz54\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:22.053421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:22.053387 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:37:22.058175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:22.058146 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2a4123-openshift-default-554c56cc99-8hsck"] Oct 30 22:37:22.146643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:22.146562 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" path="/var/lib/kubelet/pods/86002a59-1b82-4516-8b86-c3251d6936cb/volumes" Oct 30 22:37:25.149086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:25.149043 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf"] Oct 30 22:37:25.199410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:25.199383 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-97aa64-5fgnf"] Oct 30 22:37:26.146586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:26.146541 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e83241-bf6b-49fc-97db-26beacc87595" path="/var/lib/kubelet/pods/c4e83241-bf6b-49fc-97db-26beacc87595/volumes" Oct 30 22:37:26.640141 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:37:26.640103 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:37:32.066546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:32.066511 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k"] Oct 30 22:37:32.116911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:32.116879 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffc4eb2-qnw4k"] Oct 30 22:37:32.146205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:32.146174 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e3ea87-67c5-451e-97ef-52612bb25713" path="/var/lib/kubelet/pods/02e3ea87-67c5-451e-97ef-52612bb25713/volumes" Oct 30 22:37:34.063546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.063514 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt"] Oct 30 22:37:34.113781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.113746 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j"] Oct 30 22:37:34.116357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.116326 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa7e7a-wddzt"] Oct 30 22:37:34.118725 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.118701 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35778301-tfr5j"] Oct 30 22:37:34.145652 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.145627 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1187f480-c592-4a8c-964f-b3dab2215144" path="/var/lib/kubelet/pods/1187f480-c592-4a8c-964f-b3dab2215144/volumes" Oct 30 22:37:34.146032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:34.146018 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f592536-365a-4c91-9a4e-913a445e9e4f" path="/var/lib/kubelet/pods/2f592536-365a-4c91-9a4e-913a445e9e4f/volumes" Oct 30 22:37:37.623428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:37.623398 4292 scope.go:122] "RemoveContainer" containerID="7b20cef4c7120eb3cb726656594355a297f6807c00be8131125739e7a5765a6c" Oct 30 22:37:37.633248 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:37.633223 4292 scope.go:122] "RemoveContainer" containerID="67b159a8ff160664a130cbe6b8ac6aac27e0af021cdb5489db71a3be56f46c90" Oct 30 22:37:37.642033 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:37:37.642010 4292 scope.go:122] "RemoveContainer" containerID="bd1ea81a7285341e52b47b403878b3b2ad136eb0f3dad4ff0e5300cee1b9cd21" Oct 30 22:37:56.640014 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:37:56.639974 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:38:21.536146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536109 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9"] Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536782 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536799 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536819 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536824 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536836 4292 cpu_manager.go:414] "containerMap: removing container" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536841 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536851 4292 cpu_manager.go:414] "containerMap: removing container" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536856 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536955 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536964 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" containerName="revoke-keys" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536972 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.538832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.536978 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="86002a59-1b82-4516-8b86-c3251d6936cb" containerName="istio-proxy" Oct 30 22:38:21.540259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.540239 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.619619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.619561 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.619777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.619700 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rscm\" (UniqueName: \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.619839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.619794 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.644785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.644756 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:21.644934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.644805 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:21.644934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.644823 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:38:21.644934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.644888 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:21.647767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.647747 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9"] Oct 30 22:38:21.720718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.720690 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.720892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.720759 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rscm\" (UniqueName: \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.720892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.720849 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.721470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.721444 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.723027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.723007 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.784483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.784451 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rscm\" (UniqueName: \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\") pod \"maas-api-revoke-sub-aaa1d8ad-9v9x9\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:21.854158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:21.854134 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:22.237464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:22.237439 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9"] Oct 30 22:38:22.238391 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:38:22.238363 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffeb67a1_bd4f_4a78_9f1a_8ca46ce3e0bf.slice/crio-3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9 WatchSource:0}: Error finding container 3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9: Status 404 returned error can't find the container with id 3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9 Oct 30 22:38:22.313653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:22.313626 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" event={"ID":"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf","Type":"ContainerStarted","Data":"3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9"} Oct 30 22:38:23.321939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:23.321859 4292 generic.go:356] "Generic (PLEG): container finished" podID="ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" containerID="648dd1d23af7d1c4543bef24d6869617da80be3aac115d6615403d62c644c666" exitCode=0 Oct 30 22:38:23.322304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:23.321942 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" event={"ID":"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf","Type":"ContainerDied","Data":"648dd1d23af7d1c4543bef24d6869617da80be3aac115d6615403d62c644c666"} Oct 30 22:38:24.468362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.468338 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:24.547748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.547720 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\" (UniqueName: \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\") pod \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " Oct 30 22:38:24.547892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.547800 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\") pod \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " Oct 30 22:38:24.547892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.547866 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\") pod \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\" (UID: \"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf\") " Oct 30 22:38:24.548601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.548523 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca" pod "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" (UID: "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:24.550197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.550171 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm" pod "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" (UID: "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf"). InnerVolumeSpecName "kube-api-access-6rscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:24.552470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.550342 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:24.559121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.559095 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca" pod "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" (UID: "ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:24.652024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.651953 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:24.652024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:24.651981 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rscm\" (UniqueName: \"kubernetes.io/projected/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf-kube-api-access-6rscm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:25.335289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:25.335254 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" Oct 30 22:38:25.335462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:25.335294 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9" event={"ID":"ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf","Type":"ContainerDied","Data":"3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9"} Oct 30 22:38:25.335462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:25.335342 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca3b55460127c8cad20c1e551ad256bd94928671f6f92f82879197fff8117c9" Oct 30 22:38:26.639869 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:38:26.639823 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:38:33.563956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:33.563919 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:38:34.558254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.558216 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv"] Oct 30 22:38:34.559345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.559314 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" containerName="revoke-keys" Oct 30 22:38:34.559509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.559495 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" containerName="revoke-keys" Oct 30 22:38:34.559799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.559781 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" containerName="revoke-keys" Oct 30 22:38:34.563703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.563670 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.621006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.620968 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv"] Oct 30 22:38:34.660241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.660212 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.660407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.660312 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.660526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.660507 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.666784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.666760 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:34.666909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.666795 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:34.666909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.666801 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:38:34.667888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.667859 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:34.761887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.761856 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.762053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.761913 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.762053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.762048 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.762681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.762663 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.766555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.766533 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.827748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.827681 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\") pod \"maas-api-revoke-sub-f073dff1-kxhlv\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:34.879375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:34.879347 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:35.260132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:35.260102 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv"] Oct 30 22:38:35.260640 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:38:35.260615 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cce9ff3_8bb7_49c4_a47a_e4fb070d3594.slice/crio-ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e WatchSource:0}: Error finding container ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e: Status 404 returned error can't find the container with id ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e Oct 30 22:38:35.395655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:35.395605 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" event={"ID":"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594","Type":"ContainerStarted","Data":"ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e"} Oct 30 22:38:36.404032 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:36.403947 4292 generic.go:356] "Generic (PLEG): container finished" podID="0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" containerID="cfe1fa5c45562a6eaafac27fd8a609d32f717dd75a4e10b71c5b8aef3cadc6ab" exitCode=0 Oct 30 22:38:36.404382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:36.404031 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" event={"ID":"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594","Type":"ContainerDied","Data":"cfe1fa5c45562a6eaafac27fd8a609d32f717dd75a4e10b71c5b8aef3cadc6ab"} Oct 30 22:38:37.555830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.555801 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:37.680363 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.680290 4292 scope.go:122] "RemoveContainer" containerID="59e70da7ac7ea891c62b4c5a4ff0cc9eeebd303ed5920fd7813e18e3ef2e5594" Oct 30 22:38:37.689785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.689762 4292 scope.go:122] "RemoveContainer" containerID="bf07db9cfc8d7bca52f5314c5cd98e3e1490bf7f81e3312d8e1103bde1c008c2" Oct 30 22:38:37.693840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.693802 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\") pod \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " Oct 30 22:38:37.693972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.693917 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\") pod \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " Oct 30 22:38:37.694074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.694055 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\") pod \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\" (UID: \"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594\") " Oct 30 22:38:37.694874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.694848 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca" pod "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" (UID: "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:37.697073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.697047 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h" pod "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" (UID: "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594"). InnerVolumeSpecName "kube-api-access-btf2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:37.701255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.701216 4292 scope.go:122] "RemoveContainer" containerID="9b507c0e2533a14ee3e57fd372c6fad36ab9f8362bd7c3291b2a98b669ce4f9a" Oct 30 22:38:37.703335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.703308 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca" pod "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" (UID: "0cce9ff3-8bb7-49c4-a47a-e4fb070d3594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:37.722915 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.722894 4292 scope.go:122] "RemoveContainer" containerID="599b8f1b070bc21990e563eeb98010c4ad637b05f8ba1ea1130d7cbd2a89f62b" Oct 30 22:38:37.733440 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.733332 4292 scope.go:122] "RemoveContainer" containerID="880317f7835224fd1706c6246a267ca49424ceebfc7e1c6479bfecbd30b8ca33" Oct 30 22:38:37.742316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.742292 4292 scope.go:122] "RemoveContainer" containerID="05799103e2c90f71574287338309fc41e61eff6545693d33801b7bc498841001" Oct 30 22:38:37.795628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.795600 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-kube-api-access-btf2h\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:37.795717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.795630 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:37.795717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:37.795648 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:38.416988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:38.416918 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" Oct 30 22:38:38.416988 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:38.416954 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv" event={"ID":"0cce9ff3-8bb7-49c4-a47a-e4fb070d3594","Type":"ContainerDied","Data":"ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e"} Oct 30 22:38:38.417170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:38.417005 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8fe04d6f1a9289275315e3b18feffbf0e04f74f7e37ea7d5550dd4fa2c693e" Oct 30 22:38:48.587339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.587229 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" podUID="74c646e7-2555-4dcf-acba-6dda556dd948" containerName="main" containerID="cri-o://2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb" gracePeriod=45 Oct 30 22:38:48.845013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.844955 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:38:48.899343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899318 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899349 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899380 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899430 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\" (UniqueName: \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899503 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899594 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\" (UniqueName: \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\") pod \"74c646e7-2555-4dcf-acba-6dda556dd948\" (UID: \"74c646e7-2555-4dcf-acba-6dda556dd948\") " Oct 30 22:38:48.899709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:48.899709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899664 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:48.899900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899717 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:48.899968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899948 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:48.899968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899962 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:48.900070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.899971 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:48.901640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.901613 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:48.901768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.901698 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "kube-api-access-8g8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:48.901768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:48.901699 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs" pod "74c646e7-2555-4dcf-acba-6dda556dd948" (UID: "74c646e7-2555-4dcf-acba-6dda556dd948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:49.000979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.000951 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c646e7-2555-4dcf-acba-6dda556dd948-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:49.000979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.000976 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8vg\" (UniqueName: \"kubernetes.io/projected/74c646e7-2555-4dcf-acba-6dda556dd948-kube-api-access-8g8vg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:49.000979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.000985 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c646e7-2555-4dcf-acba-6dda556dd948-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:49.483275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.483240 4292 generic.go:356] "Generic (PLEG): container finished" podID="74c646e7-2555-4dcf-acba-6dda556dd948" containerID="2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb" exitCode=0 Oct 30 22:38:49.483437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.483286 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" Oct 30 22:38:49.483437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.483323 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" event={"ID":"74c646e7-2555-4dcf-acba-6dda556dd948","Type":"ContainerDied","Data":"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb"} Oct 30 22:38:49.483437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.483369 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4" event={"ID":"74c646e7-2555-4dcf-acba-6dda556dd948","Type":"ContainerDied","Data":"a0a94291de4a9bdbfde7b68b5af88eae0866538495b58522d086b2e310ccc08a"} Oct 30 22:38:49.483437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.483391 4292 scope.go:122] "RemoveContainer" containerID="2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb" Oct 30 22:38:49.493152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.493134 4292 scope.go:122] "RemoveContainer" containerID="2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb" Oct 30 22:38:49.493427 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:38:49.493406 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb\": container with ID starting with 2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb not found: ID does not exist" containerID="2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb" Oct 30 22:38:49.493499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.493439 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb"} err="failed to get container status \"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb\": rpc error: code = NotFound desc = could not find container \"2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb\": container with ID starting with 2125962330a89b1e2f7e20f3df5a8087d6a4d7bbe504a1d59fe24e22bd673adb not found: ID does not exist" Oct 30 22:38:49.655051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.655021 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:38:49.660081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:49.660059 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cd5407/test-model-cd5407-kserve-599df5f9fc-pr9q4"] Oct 30 22:38:50.149530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:50.149493 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c646e7-2555-4dcf-acba-6dda556dd948" path="/var/lib/kubelet/pods/74c646e7-2555-4dcf-acba-6dda556dd948/volumes" Oct 30 22:38:52.982902 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:52.982867 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:38:53.540761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.540731 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw"] Oct 30 22:38:53.541425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541406 4292 cpu_manager.go:414] "containerMap: removing container" podUID="74c646e7-2555-4dcf-acba-6dda556dd948" containerName="main" Oct 30 22:38:53.541492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541429 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c646e7-2555-4dcf-acba-6dda556dd948" containerName="main" Oct 30 22:38:53.541530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541498 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" containerName="revoke-keys" Oct 30 22:38:53.541530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541506 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" containerName="revoke-keys" Oct 30 22:38:53.541656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541643 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" containerName="revoke-keys" Oct 30 22:38:53.541705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.541663 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c646e7-2555-4dcf-acba-6dda556dd948" containerName="main" Oct 30 22:38:53.546474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.546453 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.603066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.603041 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:53.603244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.603108 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:53.603244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.603218 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:38:53.603386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.603284 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:53.607457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.607433 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw"] Oct 30 22:38:53.649535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.649511 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.649664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.649552 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfdz8\" (UniqueName: \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.649710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.649683 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.750664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.750637 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.750786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.750678 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfdz8\" (UniqueName: \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.750786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.750722 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.751403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.751382 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.752982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.752953 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.814640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.814614 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfdz8\" (UniqueName: \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\") pod \"maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:53.860329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:53.860303 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:54.238151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:54.238116 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw"] Oct 30 22:38:54.238515 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:38:54.238436 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4200fa9d_f220_466d_bfd0_957d03777bd5.slice/crio-6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a WatchSource:0}: Error finding container 6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a: Status 404 returned error can't find the container with id 6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a Oct 30 22:38:54.514135 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:54.514103 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" event={"ID":"4200fa9d-f220-466d-bfd0-957d03777bd5","Type":"ContainerStarted","Data":"8624bbbe942780ac0987638dcf8313ac5088f03b3b4389dadb33a8e7c4344978"} Oct 30 22:38:54.514270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:54.514159 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" event={"ID":"4200fa9d-f220-466d-bfd0-957d03777bd5","Type":"ContainerStarted","Data":"6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a"} Oct 30 22:38:54.636108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:54.636057 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" podStartSLOduration=1.636043723 podStartE2EDuration="1.636043723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:54.634648195 +0000 UTC m=+2662.603467761" watchObservedRunningTime="2026-10-30 22:38:54.636043723 +0000 UTC m=+2662.604863329" Oct 30 22:38:55.521036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:55.520943 4292 generic.go:356] "Generic (PLEG): container finished" podID="4200fa9d-f220-466d-bfd0-957d03777bd5" containerID="8624bbbe942780ac0987638dcf8313ac5088f03b3b4389dadb33a8e7c4344978" exitCode=0 Oct 30 22:38:55.521445 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:55.521030 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" event={"ID":"4200fa9d-f220-466d-bfd0-957d03777bd5","Type":"ContainerDied","Data":"8624bbbe942780ac0987638dcf8313ac5088f03b3b4389dadb33a8e7c4344978"} Oct 30 22:38:56.640426 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:38:56.640385 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:38:56.668672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.668650 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:56.784236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.784179 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\") pod \"4200fa9d-f220-466d-bfd0-957d03777bd5\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " Oct 30 22:38:56.784236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.784213 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\") pod \"4200fa9d-f220-466d-bfd0-957d03777bd5\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " Oct 30 22:38:56.784403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.784304 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\" (UniqueName: \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\") pod \"4200fa9d-f220-466d-bfd0-957d03777bd5\" (UID: \"4200fa9d-f220-466d-bfd0-957d03777bd5\") " Oct 30 22:38:56.784506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.784485 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca" pod "4200fa9d-f220-466d-bfd0-957d03777bd5" (UID: "4200fa9d-f220-466d-bfd0-957d03777bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:56.784722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.784700 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:56.786358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.786332 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8" pod "4200fa9d-f220-466d-bfd0-957d03777bd5" (UID: "4200fa9d-f220-466d-bfd0-957d03777bd5"). InnerVolumeSpecName "kube-api-access-wfdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:56.786358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.786346 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca" pod "4200fa9d-f220-466d-bfd0-957d03777bd5" (UID: "4200fa9d-f220-466d-bfd0-957d03777bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:56.885669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.885637 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfdz8\" (UniqueName: \"kubernetes.io/projected/4200fa9d-f220-466d-bfd0-957d03777bd5-kube-api-access-wfdz8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:56.885669 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:56.885663 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4200fa9d-f220-466d-bfd0-957d03777bd5-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:57.534873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:57.534836 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" Oct 30 22:38:57.534873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:57.534863 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw" event={"ID":"4200fa9d-f220-466d-bfd0-957d03777bd5","Type":"ContainerDied","Data":"6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a"} Oct 30 22:38:57.535106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:38:57.534916 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a61bd80cec5d5c3bcae03b8555a739e2068a357c68240250d7436de69555c1a" Oct 30 22:39:05.034061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:05.034018 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:39:05.143668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:05.143637 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:39:08.002889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.002835 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" podUID="8a80b571-ce37-4ec8-ade7-80e670b49be2" containerName="main" containerID="cri-o://0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0" gracePeriod=45 Oct 30 22:39:08.265321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.265263 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:39:08.409160 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409120 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409191 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409242 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409290 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\" (UniqueName: \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409328 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409411 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\") pod \"8a80b571-ce37-4ec8-ade7-80e670b49be2\" (UID: \"8a80b571-ce37-4ec8-ade7-80e670b49be2\") " Oct 30 22:39:08.409603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409476 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.409762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409737 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.409918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409896 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.409995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.409779 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.411544 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.411516 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.411685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.411611 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "kube-api-access-lszrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:08.411856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.411839 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs" pod "8a80b571-ce37-4ec8-ade7-80e670b49be2" (UID: "8a80b571-ce37-4ec8-ade7-80e670b49be2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:08.511395 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.511367 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.511493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.511397 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lszrg\" (UniqueName: \"kubernetes.io/projected/8a80b571-ce37-4ec8-ade7-80e670b49be2-kube-api-access-lszrg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.511493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.511411 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.511493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.511425 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b571-ce37-4ec8-ade7-80e670b49be2-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.511493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.511437 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b571-ce37-4ec8-ade7-80e670b49be2-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.600665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.600632 4292 generic.go:356] "Generic (PLEG): container finished" podID="8a80b571-ce37-4ec8-ade7-80e670b49be2" containerID="0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0" exitCode=0 Oct 30 22:39:08.600795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.600671 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" Oct 30 22:39:08.600795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.600720 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" event={"ID":"8a80b571-ce37-4ec8-ade7-80e670b49be2","Type":"ContainerDied","Data":"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0"} Oct 30 22:39:08.600795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.600768 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn" event={"ID":"8a80b571-ce37-4ec8-ade7-80e670b49be2","Type":"ContainerDied","Data":"1fc227c921b313676a9da35a8206f6105b825ba97e620c5e7e233bbe764fc7c3"} Oct 30 22:39:08.600795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.600789 4292 scope.go:122] "RemoveContainer" containerID="0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0" Oct 30 22:39:08.610462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.610443 4292 scope.go:122] "RemoveContainer" containerID="0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0" Oct 30 22:39:08.610747 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:08.610729 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0\": container with ID starting with 0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0 not found: ID does not exist" containerID="0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0" Oct 30 22:39:08.610830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.610754 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0"} err="failed to get container status \"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0\": rpc error: code = NotFound desc = could not find container \"0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0\": container with ID starting with 0c506eaeca22f1ce23676d10a083677d316611322ec6fcbdcc4472a7390f67c0 not found: ID does not exist" Oct 30 22:39:08.770455 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.770426 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:39:08.776484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:08.776462 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e54161/test-model-e54161-kserve-5bf748544c-5xcfn"] Oct 30 22:39:10.034670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.034621 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" podUID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" containerName="payload-processing" containerID="cri-o://2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf" gracePeriod=25 Oct 30 22:39:10.144536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.144485 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" containerID="cri-o://5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da" gracePeriod=25 Oct 30 22:39:10.148001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.147969 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a80b571-ce37-4ec8-ade7-80e670b49be2" path="/var/lib/kubelet/pods/8a80b571-ce37-4ec8-ade7-80e670b49be2/volumes" Oct 30 22:39:10.295094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.295066 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:39:10.440052 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.440021 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\" (UniqueName: \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\") pod \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " Oct 30 22:39:10.440205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.440084 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\") pod \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\" (UID: \"2c8bfe25-3e44-4209-ae79-c5dbc179fd25\") " Oct 30 22:39:10.440532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.440498 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume" pod "2c8bfe25-3e44-4209-ae79-c5dbc179fd25" (UID: "2c8bfe25-3e44-4209-ae79-c5dbc179fd25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:10.442091 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.442068 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74" pod "2c8bfe25-3e44-4209-ae79-c5dbc179fd25" (UID: "2c8bfe25-3e44-4209-ae79-c5dbc179fd25"). InnerVolumeSpecName "kube-api-access-c9b74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:10.541419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.541387 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9b74\" (UniqueName: \"kubernetes.io/projected/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-kube-api-access-c9b74\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:10.541419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.541413 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8bfe25-3e44-4209-ae79-c5dbc179fd25-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:10.616067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.615979 4292 generic.go:356] "Generic (PLEG): container finished" podID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" containerID="2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf" exitCode=0 Oct 30 22:39:10.616067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.616027 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" Oct 30 22:39:10.616067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.616057 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" event={"ID":"2c8bfe25-3e44-4209-ae79-c5dbc179fd25","Type":"ContainerDied","Data":"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf"} Oct 30 22:39:10.616309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.616112 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7" event={"ID":"2c8bfe25-3e44-4209-ae79-c5dbc179fd25","Type":"ContainerDied","Data":"7552a82bc2260adb89476c122841ea8692a88e282c2972d5120c7580405994da"} Oct 30 22:39:10.616309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.616137 4292 scope.go:122] "RemoveContainer" containerID="2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf" Oct 30 22:39:10.625647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.625630 4292 scope.go:122] "RemoveContainer" containerID="2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf" Oct 30 22:39:10.625920 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:10.625901 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf\": container with ID starting with 2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf not found: ID does not exist" containerID="2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf" Oct 30 22:39:10.625976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.625928 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf"} err="failed to get container status \"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf\": rpc error: code = NotFound desc = could not find container \"2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf\": container with ID starting with 2137eedf5dd224f82c60db4505bce6329202b5a14953b1e9ef2a8e469ee039bf not found: ID does not exist" Oct 30 22:39:10.786017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.785984 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:39:10.791476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:10.791452 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd5407-7595fcd75-5pcv7"] Oct 30 22:39:12.147220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:12.147188 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" path="/var/lib/kubelet/pods/2c8bfe25-3e44-4209-ae79-c5dbc179fd25/volumes" Oct 30 22:39:12.254784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:12.254754 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:39:13.056629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:13.056594 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl"] Oct 30 22:39:13.108597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:13.108543 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823eee4e-25xxl"] Oct 30 22:39:14.145685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:14.145656 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ec5de6-476a-4202-8857-b009189837c5" path="/var/lib/kubelet/pods/33ec5de6-476a-4202-8857-b009189837c5/volumes" Oct 30 22:39:22.254793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:22.254749 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:39:23.925340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:23.925301 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw"] Oct 30 22:39:23.975616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:23.975588 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd5407-bjfpw"] Oct 30 22:39:24.145668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:24.145639 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4200fa9d-f220-466d-bfd0-957d03777bd5" path="/var/lib/kubelet/pods/4200fa9d-f220-466d-bfd0-957d03777bd5/volumes" Oct 30 22:39:26.639985 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:26.639938 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:39:30.402778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.402721 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:39:30.452900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.452877 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\" (UniqueName: \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\") pod \"636da51d-84f5-4495-a50f-f2c3b78d5680\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " Oct 30 22:39:30.453014 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.452918 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\") pod \"636da51d-84f5-4495-a50f-f2c3b78d5680\" (UID: \"636da51d-84f5-4495-a50f-f2c3b78d5680\") " Oct 30 22:39:30.453357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.453326 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume" pod "636da51d-84f5-4495-a50f-f2c3b78d5680" (UID: "636da51d-84f5-4495-a50f-f2c3b78d5680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:30.454913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.454894 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79" pod "636da51d-84f5-4495-a50f-f2c3b78d5680" (UID: "636da51d-84f5-4495-a50f-f2c3b78d5680"). InnerVolumeSpecName "kube-api-access-qwm79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:30.554022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.553996 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwm79\" (UniqueName: \"kubernetes.io/projected/636da51d-84f5-4495-a50f-f2c3b78d5680-kube-api-access-qwm79\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:30.554022 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.554020 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636da51d-84f5-4495-a50f-f2c3b78d5680-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:30.744085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.744009 4292 generic.go:356] "Generic (PLEG): container finished" podID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerID="5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da" exitCode=0 Oct 30 22:39:30.744085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.744051 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" Oct 30 22:39:30.744085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.744083 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" event={"ID":"636da51d-84f5-4495-a50f-f2c3b78d5680","Type":"ContainerDied","Data":"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da"} Oct 30 22:39:30.744307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.744125 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm" event={"ID":"636da51d-84f5-4495-a50f-f2c3b78d5680","Type":"ContainerDied","Data":"72f10fc890195c2a22e9579701672e119dbedd22619f776f25c7d6895e421720"} Oct 30 22:39:30.744307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.744149 4292 scope.go:122] "RemoveContainer" containerID="5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da" Oct 30 22:39:30.757761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.757737 4292 scope.go:122] "RemoveContainer" containerID="5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da" Oct 30 22:39:30.758000 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:30.757980 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da\": container with ID starting with 5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da not found: ID does not exist" containerID="5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da" Oct 30 22:39:30.758051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.758006 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da"} err="failed to get container status \"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da\": rpc error: code = NotFound desc = could not find container \"5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da\": container with ID starting with 5de22bcc8db6d0db11e81943ca8e1c209e98fd2325b2ac0c7ee7b98bcf1db0da not found: ID does not exist" Oct 30 22:39:30.914931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.914896 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:39:30.920063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:30.920041 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd5407-7b448578bc-dtkcm"] Oct 30 22:39:32.147977 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:32.147945 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" path="/var/lib/kubelet/pods/636da51d-84f5-4495-a50f-f2c3b78d5680/volumes" Oct 30 22:39:32.285030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:32.284984 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:39:32.285210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:32.285188 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" containerID="cri-o://2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" gracePeriod=30 Oct 30 22:39:33.194364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.194322 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz"] Oct 30 22:39:33.195002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.194979 4292 cpu_manager.go:414] "containerMap: removing container" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195002 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195030 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" containerName="payload-processing" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195038 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" containerName="payload-processing" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4200fa9d-f220-466d-bfd0-957d03777bd5" containerName="revoke-keys" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195059 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4200fa9d-f220-466d-bfd0-957d03777bd5" containerName="revoke-keys" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195099 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8a80b571-ce37-4ec8-ade7-80e670b49be2" containerName="main" Oct 30 22:39:33.195113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195107 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a80b571-ce37-4ec8-ade7-80e670b49be2" containerName="main" Oct 30 22:39:33.195474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195224 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a80b571-ce37-4ec8-ade7-80e670b49be2" containerName="main" Oct 30 22:39:33.195474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195244 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="636da51d-84f5-4495-a50f-f2c3b78d5680" containerName="payload-pre-processing" Oct 30 22:39:33.195474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195255 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8bfe25-3e44-4209-ae79-c5dbc179fd25" containerName="payload-processing" Oct 30 22:39:33.195474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.195264 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4200fa9d-f220-466d-bfd0-957d03777bd5" containerName="revoke-keys" Oct 30 22:39:33.198713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.198691 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.258197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.258172 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz"] Oct 30 22:39:33.284112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.284081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.284308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.284280 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.284393 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.284325 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlp8v\" (UniqueName: \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.304535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.304510 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:39:33.304535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.304523 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:39:33.304719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.304538 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:39:33.304719 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.304632 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:39:33.385866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.385840 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.385975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.385925 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.385975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.385945 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlp8v\" (UniqueName: \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.386561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.386536 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.388116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.388099 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.449671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.449616 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlp8v\" (UniqueName: \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\") pod \"maas-api-revoke-keys-e2e-inf-b-e54161-b28bz\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.511317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.511293 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:33.886085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:33.886054 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz"] Oct 30 22:39:33.886216 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:39:33.886134 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a762795_40ae_45b9_9ca2_72ea3e636952.slice/crio-c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031 WatchSource:0}: Error finding container c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031: Status 404 returned error can't find the container with id c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031 Oct 30 22:39:34.771838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:34.771810 4292 generic.go:356] "Generic (PLEG): container finished" podID="3a762795-40ae-45b9-9ca2-72ea3e636952" containerID="6740fe8a563370f06b00bc8caa1e5ed2abd86710c326ab62ca7f59bc36c0b886" exitCode=0 Oct 30 22:39:34.772158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:34.771903 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" event={"ID":"3a762795-40ae-45b9-9ca2-72ea3e636952","Type":"ContainerDied","Data":"6740fe8a563370f06b00bc8caa1e5ed2abd86710c326ab62ca7f59bc36c0b886"} Oct 30 22:39:34.772158 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:34.771956 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" event={"ID":"3a762795-40ae-45b9-9ca2-72ea3e636952","Type":"ContainerStarted","Data":"c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031"} Oct 30 22:39:35.919844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:35.919817 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:35.948373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:35.948337 4292 patch_prober.go:29] interesting pod/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:39:35.948547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:35.948391 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:39:36.012956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.012931 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\") pod \"3a762795-40ae-45b9-9ca2-72ea3e636952\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " Oct 30 22:39:36.013092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.013001 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\") pod \"3a762795-40ae-45b9-9ca2-72ea3e636952\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " Oct 30 22:39:36.013169 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.013103 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\" (UniqueName: \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\") pod \"3a762795-40ae-45b9-9ca2-72ea3e636952\" (UID: \"3a762795-40ae-45b9-9ca2-72ea3e636952\") " Oct 30 22:39:36.013311 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.013291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca" pod "3a762795-40ae-45b9-9ca2-72ea3e636952" (UID: "3a762795-40ae-45b9-9ca2-72ea3e636952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:36.013531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.013513 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.015103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.015080 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v" pod "3a762795-40ae-45b9-9ca2-72ea3e636952" (UID: "3a762795-40ae-45b9-9ca2-72ea3e636952"). InnerVolumeSpecName "kube-api-access-mlp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:36.015222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.015174 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca" pod "3a762795-40ae-45b9-9ca2-72ea3e636952" (UID: "3a762795-40ae-45b9-9ca2-72ea3e636952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:36.114819 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.114790 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a762795-40ae-45b9-9ca2-72ea3e636952-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.114970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.114822 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlp8v\" (UniqueName: \"kubernetes.io/projected/3a762795-40ae-45b9-9ca2-72ea3e636952-kube-api-access-mlp8v\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.784398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.784369 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" Oct 30 22:39:36.784557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.784392 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz" event={"ID":"3a762795-40ae-45b9-9ca2-72ea3e636952","Type":"ContainerDied","Data":"c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031"} Oct 30 22:39:36.784557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:36.784445 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37f15e263ee89b6ada4a718329de098d8c66dfdf85fab29dcad98e8a8a2c031" Oct 30 22:39:37.538919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.538894 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:39:37.637434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637375 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637560 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637444 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637626 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637690 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637759 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637790 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.637909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637803 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:37.637909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637845 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.638097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637942 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.638097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637957 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:37.638097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637981 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:37.638097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.637988 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\") pod \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\" (UID: \"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a\") " Oct 30 22:39:37.638097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638013 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:37.638357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638131 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:37.638475 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638451 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.638541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638482 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.638541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638497 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.638541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638511 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.638541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.638526 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.639909 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.639886 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:39:37.639994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.639933 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:37.639994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.639978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:37.640074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.640063 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx" pod "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" (UID: "37c5d61c-1f0a-406f-aae6-0d9e4841ce2a"). InnerVolumeSpecName "kube-api-access-zhdzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:37.739845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.739818 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.739845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.739844 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.740028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.739860 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdzx\" (UniqueName: \"kubernetes.io/projected/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-kube-api-access-zhdzx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.740028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.739874 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.783027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.782996 4292 scope.go:122] "RemoveContainer" containerID="97a1ca69647679aaab70ae165ee2c33e71e215800c29b70c786866670d773134" Oct 30 22:39:37.792412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.792384 4292 generic.go:356] "Generic (PLEG): container finished" podID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" exitCode=0 Oct 30 22:39:37.792535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.792453 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" Oct 30 22:39:37.792535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.792462 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerDied","Data":"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b"} Oct 30 22:39:37.792673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.792548 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg" event={"ID":"37c5d61c-1f0a-406f-aae6-0d9e4841ce2a","Type":"ContainerDied","Data":"cbd1625d0c6736e23a7196940f1f63f0712c91447f69df62f68a9db34ead62a9"} Oct 30 22:39:37.792673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.792597 4292 scope.go:122] "RemoveContainer" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" Oct 30 22:39:37.793487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.793469 4292 scope.go:122] "RemoveContainer" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" Oct 30 22:39:37.805769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.805745 4292 scope.go:122] "RemoveContainer" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" Oct 30 22:39:37.806152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806136 4292 scope.go:122] "RemoveContainer" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" Oct 30 22:39:37.806421 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:37.806402 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d\": container with ID starting with b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d not found: ID does not exist" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" Oct 30 22:39:37.806495 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:37.806410 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b\": container with ID starting with 2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b not found: ID does not exist" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" Oct 30 22:39:37.806495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806434 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d"} err="failed to get container status \"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d\": rpc error: code = NotFound desc = could not find container \"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d\": container with ID starting with b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d not found: ID does not exist" Oct 30 22:39:37.806495 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806451 4292 scope.go:122] "RemoveContainer" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" Oct 30 22:39:37.806495 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:37.806441 4292 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b\": rpc error: code = NotFound desc = could not find container \"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b\": container with ID starting with 2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b not found: ID does not exist" containerID="2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b" Oct 30 22:39:37.806752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806733 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b"} err="failed to get container status \"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b\": rpc error: code = NotFound desc = could not find container \"2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b\": container with ID starting with 2b64908b8a83dd33084340fea8c32988092fb68fa959c9078e34399ffba3cd2b not found: ID does not exist" Oct 30 22:39:37.806801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806752 4292 scope.go:122] "RemoveContainer" containerID="b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d" Oct 30 22:39:37.806957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.806941 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d"} err="failed to get container status \"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d\": rpc error: code = NotFound desc = could not find container \"b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d\": container with ID starting with b2b67d84fe0e0c0551c673e0be483d3ca85d12b418d0e0a98c6b6e20cf1ae00d not found: ID does not exist" Oct 30 22:39:37.964700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.964642 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:39:37.970481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:37.970459 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cd5407-openshift-default-7c7b77988f-fsphg"] Oct 30 22:39:38.146879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:38.146850 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" path="/var/lib/kubelet/pods/37c5d61c-1f0a-406f-aae6-0d9e4841ce2a/volumes" Oct 30 22:39:44.703304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:44.703269 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:39:44.813287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:44.813254 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:39:49.704339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:49.704277 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" podUID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" containerName="payload-processing" containerID="cri-o://8f4e3f33a94e5a24cdc64cf87e7cf442ee9ecc460bacd626d7b297ebc87c06d7" gracePeriod=25 Oct 30 22:39:49.814487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:49.814437 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" containerID="cri-o://ec76d77a052a0900512f3fc71cc6b7b87d964c7b19c0cc48748c81fba68062ae" gracePeriod=25 Oct 30 22:39:49.864426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:49.864389 4292 generic.go:356] "Generic (PLEG): container finished" podID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" containerID="8f4e3f33a94e5a24cdc64cf87e7cf442ee9ecc460bacd626d7b297ebc87c06d7" exitCode=0 Oct 30 22:39:49.864426 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:49.864420 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" event={"ID":"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a","Type":"ContainerDied","Data":"8f4e3f33a94e5a24cdc64cf87e7cf442ee9ecc460bacd626d7b297ebc87c06d7"} Oct 30 22:39:49.958188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:49.958129 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:39:50.053843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.053814 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\" (UniqueName: \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\") pod \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " Oct 30 22:39:50.053980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.053888 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\") pod \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\" (UID: \"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a\") " Oct 30 22:39:50.054253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.054230 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume" pod "2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" (UID: "2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:50.055951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.055929 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9" pod "2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" (UID: "2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a"). InnerVolumeSpecName "kube-api-access-lgnb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:50.155781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.155754 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnb9\" (UniqueName: \"kubernetes.io/projected/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-kube-api-access-lgnb9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:50.155781 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.155778 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:50.260755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.260679 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:39:50.871023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.870988 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" Oct 30 22:39:50.871023 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.871015 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx" event={"ID":"2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a","Type":"ContainerDied","Data":"fa8fa2ab1138518cad1aa0b0740610d34c50e89117ab302adf354757be38a5c3"} Oct 30 22:39:50.871525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:50.871078 4292 scope.go:122] "RemoveContainer" containerID="8f4e3f33a94e5a24cdc64cf87e7cf442ee9ecc460bacd626d7b297ebc87c06d7" Oct 30 22:39:51.042058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:51.042031 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:39:51.092275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:51.092250 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e54161-74d8458447-j45vx"] Oct 30 22:39:52.145844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:52.145813 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" path="/var/lib/kubelet/pods/2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a/volumes" Oct 30 22:39:56.300557 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:56.300523 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz"] Oct 30 22:39:56.350246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:56.350209 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e54161-b28bz"] Oct 30 22:39:56.639479 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:39:56.639405 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:39:58.146086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:58.146053 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a762795-40ae-45b9-9ca2-72ea3e636952" path="/var/lib/kubelet/pods/3a762795-40ae-45b9-9ca2-72ea3e636952/volumes" Oct 30 22:39:59.057925 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:59.057889 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc"] Oct 30 22:39:59.108713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:59.108675 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p"] Oct 30 22:39:59.110931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:59.110914 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4acb8-6qdwc"] Oct 30 22:39:59.112833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:39:59.112814 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69fa0f9-5tp5p"] Oct 30 22:40:00.145347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:00.145313 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2483e519-df79-4c28-ab9b-597061dd78ea" path="/var/lib/kubelet/pods/2483e519-df79-4c28-ab9b-597061dd78ea/volumes" Oct 30 22:40:00.145776 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:00.145763 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568f2838-6128-4372-96e8-b1283305a48d" path="/var/lib/kubelet/pods/568f2838-6128-4372-96e8-b1283305a48d/volumes" Oct 30 22:40:00.261232 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:00.261191 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:02.974150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:02.974110 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:40:02.974678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:02.974354 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" containerID="cri-o://05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6" gracePeriod=30 Oct 30 22:40:05.730062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730031 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht"] Oct 30 22:40:05.730484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730471 4292 cpu_manager.go:414] "containerMap: removing container" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:05.730533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730485 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:05.730533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730505 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" containerName="payload-processing" Oct 30 22:40:05.730533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730510 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" containerName="payload-processing" Oct 30 22:40:05.730533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730521 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3a762795-40ae-45b9-9ca2-72ea3e636952" containerName="revoke-keys" Oct 30 22:40:05.730533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730526 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a762795-40ae-45b9-9ca2-72ea3e636952" containerName="revoke-keys" Oct 30 22:40:05.730721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730542 4292 cpu_manager.go:414] "containerMap: removing container" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:05.730721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730548 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:05.730721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730637 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce95e8f-bf13-4dcf-a865-b3f472f6dc5a" containerName="payload-processing" Oct 30 22:40:05.730721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730648 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a762795-40ae-45b9-9ca2-72ea3e636952" containerName="revoke-keys" Oct 30 22:40:05.730721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.730657 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:05.733761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.733741 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.792554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.792528 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht"] Oct 30 22:40:05.822168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.822135 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.822292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.822188 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgbq\" (UniqueName: \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.822341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.822280 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.840172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.840145 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:05.840287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.840209 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:05.840287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.840211 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:05.840287 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.840240 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:40:05.923627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.923604 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.923730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.923641 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgbq\" (UniqueName: \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.923787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.923763 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.924231 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.924213 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.926221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.926196 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:05.987360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:05.987293 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgbq\" (UniqueName: \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:06.046330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:06.046302 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:06.423762 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:06.423727 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht"] Oct 30 22:40:06.425414 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:40:06.425390 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ab43e9_c995_419c_a905_42dd662a9894.slice/crio-5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed WatchSource:0}: Error finding container 5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed: Status 404 returned error can't find the container with id 5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed Oct 30 22:40:06.960488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:06.960415 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" event={"ID":"49ab43e9-c995-419c-a905-42dd662a9894","Type":"ContainerStarted","Data":"7ee56c17d074d9f2bb62ca3416446ffd67389c12b283bdeb0407d9b3c90c853b"} Oct 30 22:40:06.960488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:06.960464 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" event={"ID":"49ab43e9-c995-419c-a905-42dd662a9894","Type":"ContainerStarted","Data":"5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed"} Oct 30 22:40:07.087168 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:07.087118 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" podStartSLOduration=2.08710556 podStartE2EDuration="2.08710556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:07.086236222 +0000 UTC m=+2735.055055801" watchObservedRunningTime="2026-10-30 22:40:07.08710556 +0000 UTC m=+2735.055925142" Oct 30 22:40:07.888387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:07.888350 4292 patch_prober.go:29] interesting pod/e2e-inf-b-e54161-openshift-default-7899496757-prk5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:40:07.888555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:07.888416 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:40:07.967881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:07.967845 4292 generic.go:356] "Generic (PLEG): container finished" podID="49ab43e9-c995-419c-a905-42dd662a9894" containerID="7ee56c17d074d9f2bb62ca3416446ffd67389c12b283bdeb0407d9b3c90c853b" exitCode=0 Oct 30 22:40:07.968216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:07.967932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" event={"ID":"49ab43e9-c995-419c-a905-42dd662a9894","Type":"ContainerDied","Data":"7ee56c17d074d9f2bb62ca3416446ffd67389c12b283bdeb0407d9b3c90c853b"} Oct 30 22:40:08.243225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.243202 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:40:08.347302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347277 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347390 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347413 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347432 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347471 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347508 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347537 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347565 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:08.347770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347673 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:08.347770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347690 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\") pod \"163dfadf-f06d-44bf-84f7-db14adcfbb82\" (UID: \"163dfadf-f06d-44bf-84f7-db14adcfbb82\") " Oct 30 22:40:08.347982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.347953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:08.348235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.348199 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:08.348235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.348223 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.348425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.348246 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.348425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.348261 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.348425 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.348392 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:08.349750 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.349720 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "kube-api-access-bz2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:08.349887 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.349868 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:08.349956 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.349894 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:08.350112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.350094 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy" pod "163dfadf-f06d-44bf-84f7-db14adcfbb82" (UID: "163dfadf-f06d-44bf-84f7-db14adcfbb82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448928 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448958 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163dfadf-f06d-44bf-84f7-db14adcfbb82-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448967 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448976 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448986 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2sc\" (UniqueName: \"kubernetes.io/projected/163dfadf-f06d-44bf-84f7-db14adcfbb82-kube-api-access-bz2sc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.448999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.448996 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163dfadf-f06d-44bf-84f7-db14adcfbb82-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.974699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.974662 4292 generic.go:356] "Generic (PLEG): container finished" podID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerID="05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6" exitCode=0 Oct 30 22:40:08.975149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.974710 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" Oct 30 22:40:08.975149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.974745 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" event={"ID":"163dfadf-f06d-44bf-84f7-db14adcfbb82","Type":"ContainerDied","Data":"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6"} Oct 30 22:40:08.975149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.974802 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m" event={"ID":"163dfadf-f06d-44bf-84f7-db14adcfbb82","Type":"ContainerDied","Data":"185f05a9b92a63f89054cef115134d5a21e2db6c392192ab7e09f11aabd92a5d"} Oct 30 22:40:08.975149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.974825 4292 scope.go:122] "RemoveContainer" containerID="05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6" Oct 30 22:40:08.985840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.985821 4292 scope.go:122] "RemoveContainer" containerID="05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6" Oct 30 22:40:08.986121 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:40:08.986102 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6\": container with ID starting with 05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6 not found: ID does not exist" containerID="05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6" Oct 30 22:40:08.986191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:08.986133 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6"} err="failed to get container status \"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6\": rpc error: code = NotFound desc = could not find container \"05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6\": container with ID starting with 05b7735341a57285f00726e3d66764a6768132af62b77d5ef390f045d3d848e6 not found: ID does not exist" Oct 30 22:40:09.104155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.104131 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:09.154996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.154971 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\") pod \"49ab43e9-c995-419c-a905-42dd662a9894\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " Oct 30 22:40:09.155100 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.155034 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\") pod \"49ab43e9-c995-419c-a905-42dd662a9894\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " Oct 30 22:40:09.155170 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.155158 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\" (UniqueName: \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\") pod \"49ab43e9-c995-419c-a905-42dd662a9894\" (UID: \"49ab43e9-c995-419c-a905-42dd662a9894\") " Oct 30 22:40:09.155352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.155325 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca" pod "49ab43e9-c995-419c-a905-42dd662a9894" (UID: "49ab43e9-c995-419c-a905-42dd662a9894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:09.155485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.155469 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:09.157008 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.156985 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca" pod "49ab43e9-c995-419c-a905-42dd662a9894" (UID: "49ab43e9-c995-419c-a905-42dd662a9894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:09.157084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.157049 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq" pod "49ab43e9-c995-419c-a905-42dd662a9894" (UID: "49ab43e9-c995-419c-a905-42dd662a9894"). InnerVolumeSpecName "kube-api-access-xsgbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:09.171935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.171915 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:40:09.222245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.222216 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e54161-openshift-default-7899496757-prk5m"] Oct 30 22:40:09.256078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.256009 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsgbq\" (UniqueName: \"kubernetes.io/projected/49ab43e9-c995-419c-a905-42dd662a9894-kube-api-access-xsgbq\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:09.256078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.256035 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ab43e9-c995-419c-a905-42dd662a9894-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:09.985730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.983925 4292 generic.go:356] "Generic (PLEG): container finished" podID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerID="ec76d77a052a0900512f3fc71cc6b7b87d964c7b19c0cc48748c81fba68062ae" exitCode=0 Oct 30 22:40:09.985730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.984000 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" event={"ID":"c319bc3b-25e7-407a-9769-d4e6be68ad00","Type":"ContainerDied","Data":"ec76d77a052a0900512f3fc71cc6b7b87d964c7b19c0cc48748c81fba68062ae"} Oct 30 22:40:09.986157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.985944 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" event={"ID":"49ab43e9-c995-419c-a905-42dd662a9894","Type":"ContainerDied","Data":"5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed"} Oct 30 22:40:09.986157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.985995 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e381cd93695ccc1229313c46b7aa7bbede26b91e1aed25e74a7126be92d74ed" Oct 30 22:40:09.986157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:09.986082 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht" Oct 30 22:40:10.052890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.052864 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:40:10.145913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.145885 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" path="/var/lib/kubelet/pods/163dfadf-f06d-44bf-84f7-db14adcfbb82/volumes" Oct 30 22:40:10.165316 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.165290 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\") pod \"c319bc3b-25e7-407a-9769-d4e6be68ad00\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " Oct 30 22:40:10.165449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.165345 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\" (UniqueName: \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\") pod \"c319bc3b-25e7-407a-9769-d4e6be68ad00\" (UID: \"c319bc3b-25e7-407a-9769-d4e6be68ad00\") " Oct 30 22:40:10.165686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.165663 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume" pod "c319bc3b-25e7-407a-9769-d4e6be68ad00" (UID: "c319bc3b-25e7-407a-9769-d4e6be68ad00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:10.167302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.167283 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d" pod "c319bc3b-25e7-407a-9769-d4e6be68ad00" (UID: "c319bc3b-25e7-407a-9769-d4e6be68ad00"). InnerVolumeSpecName "kube-api-access-zgf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:10.266851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.266817 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c319bc3b-25e7-407a-9769-d4e6be68ad00-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:10.266851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.266846 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf7d\" (UniqueName: \"kubernetes.io/projected/c319bc3b-25e7-407a-9769-d4e6be68ad00-kube-api-access-zgf7d\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:10.993315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.993281 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" Oct 30 22:40:10.993809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.993308 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc" event={"ID":"c319bc3b-25e7-407a-9769-d4e6be68ad00","Type":"ContainerDied","Data":"bb8e07a6ee25097984780b73390d3f689ae5659d140d80509574fa510e7284bb"} Oct 30 22:40:10.993809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:10.993391 4292 scope.go:122] "RemoveContainer" containerID="ec76d77a052a0900512f3fc71cc6b7b87d964c7b19c0cc48748c81fba68062ae" Oct 30 22:40:11.175927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:11.175885 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:40:11.226507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:11.226473 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e54161-6484f4bd87-6ttqc"] Oct 30 22:40:12.146176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:12.146144 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" path="/var/lib/kubelet/pods/c319bc3b-25e7-407a-9769-d4e6be68ad00/volumes" Oct 30 22:40:21.934551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:21.934514 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht"] Oct 30 22:40:21.984670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:21.984640 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d50b13-gfbht"] Oct 30 22:40:22.145365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:22.145336 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ab43e9-c995-419c-a905-42dd662a9894" path="/var/lib/kubelet/pods/49ab43e9-c995-419c-a905-42dd662a9894/volumes" Oct 30 22:40:26.639335 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:40:26.639294 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:40:31.083386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083330 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h"] Oct 30 22:40:31.083870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083819 4292 cpu_manager.go:414] "containerMap: removing container" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" Oct 30 22:40:31.083870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083830 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" Oct 30 22:40:31.083870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083850 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" Oct 30 22:40:31.083870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083856 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" Oct 30 22:40:31.083870 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083872 4292 cpu_manager.go:414] "containerMap: removing container" podUID="49ab43e9-c995-419c-a905-42dd662a9894" containerName="revoke-keys" Oct 30 22:40:31.084043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083877 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ab43e9-c995-419c-a905-42dd662a9894" containerName="revoke-keys" Oct 30 22:40:31.084043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083938 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="163dfadf-f06d-44bf-84f7-db14adcfbb82" containerName="istio-proxy" Oct 30 22:40:31.084043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083950 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c319bc3b-25e7-407a-9769-d4e6be68ad00" containerName="payload-pre-processing" Oct 30 22:40:31.084043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083959 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ab43e9-c995-419c-a905-42dd662a9894" containerName="revoke-keys" Oct 30 22:40:31.084043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.083967 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c5d61c-1f0a-406f-aae6-0d9e4841ce2a" containerName="istio-proxy" Oct 30 22:40:31.088713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.088693 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.147074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.147046 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h"] Oct 30 22:40:31.195094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.195062 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:31.195271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.195144 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:40:31.195271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.195169 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:31.195271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.195214 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:31.262048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.262014 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tx8m\" (UniqueName: \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.262272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.262252 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.262875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.262850 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.363868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.363798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.363868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.363861 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tx8m\" (UniqueName: \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.364056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.363971 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.364679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.364656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.366085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.366063 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.427561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.427532 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tx8m\" (UniqueName: \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:31.701979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:31.701902 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:32.081367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:32.081336 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h"] Oct 30 22:40:32.086163 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:40:32.083956 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145f5063_c447_4196_b964_7ed5c0cf37fe.slice/crio-dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181 WatchSource:0}: Error finding container dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181: Status 404 returned error can't find the container with id dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181 Oct 30 22:40:32.122111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:32.122088 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" event={"ID":"145f5063-c447-4196-b964-7ed5c0cf37fe","Type":"ContainerStarted","Data":"dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181"} Oct 30 22:40:33.130528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:33.130452 4292 generic.go:356] "Generic (PLEG): container finished" podID="145f5063-c447-4196-b964-7ed5c0cf37fe" containerID="301f6d02efeb1dcaaf4be0dd0bfa0f7ea877efcd8095236c894cc22b58aa7d67" exitCode=0 Oct 30 22:40:33.130528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:33.130500 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" event={"ID":"145f5063-c447-4196-b964-7ed5c0cf37fe","Type":"ContainerDied","Data":"301f6d02efeb1dcaaf4be0dd0bfa0f7ea877efcd8095236c894cc22b58aa7d67"} Oct 30 22:40:34.284483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.284458 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:34.394619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.394551 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\" (UniqueName: \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\") pod \"145f5063-c447-4196-b964-7ed5c0cf37fe\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " Oct 30 22:40:34.394619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.394612 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\") pod \"145f5063-c447-4196-b964-7ed5c0cf37fe\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " Oct 30 22:40:34.394775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.394702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\") pod \"145f5063-c447-4196-b964-7ed5c0cf37fe\" (UID: \"145f5063-c447-4196-b964-7ed5c0cf37fe\") " Oct 30 22:40:34.395028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.395005 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca" pod "145f5063-c447-4196-b964-7ed5c0cf37fe" (UID: "145f5063-c447-4196-b964-7ed5c0cf37fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:34.395111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.395097 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:34.396647 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.396626 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca" pod "145f5063-c447-4196-b964-7ed5c0cf37fe" (UID: "145f5063-c447-4196-b964-7ed5c0cf37fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:34.396764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.396743 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m" pod "145f5063-c447-4196-b964-7ed5c0cf37fe" (UID: "145f5063-c447-4196-b964-7ed5c0cf37fe"). InnerVolumeSpecName "kube-api-access-4tx8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:34.496479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.496450 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f5063-c447-4196-b964-7ed5c0cf37fe-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:34.496479 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:34.496476 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tx8m\" (UniqueName: \"kubernetes.io/projected/145f5063-c447-4196-b964-7ed5c0cf37fe-kube-api-access-4tx8m\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:35.142818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:35.142785 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" Oct 30 22:40:35.142818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:35.142801 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h" event={"ID":"145f5063-c447-4196-b964-7ed5c0cf37fe","Type":"ContainerDied","Data":"dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181"} Oct 30 22:40:35.143039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:35.142840 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbb481ee5d7f987d662fe8aee2b497c7fca45bc82380efbd24d46fc1e56fc181" Oct 30 22:40:37.841439 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:37.841408 4292 scope.go:122] "RemoveContainer" containerID="aa384c0bf1d5494e7a3e1cadaffd35180f91179ad5eb7619b180ca93d420583f" Oct 30 22:40:42.587942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:42.587902 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:40:42.697876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:42.697844 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:40:47.588362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.588298 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" podUID="89aa5a36-166d-4e7c-a318-ae75b9d90085" containerName="payload-processing" containerID="cri-o://c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4" gracePeriod=25 Oct 30 22:40:47.698582 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.698521 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" podUID="3a804b4b-4475-47fa-9181-acf2327826e3" containerName="payload-pre-processing" containerID="cri-o://d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623" gracePeriod=25 Oct 30 22:40:47.859336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.859309 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:40:47.923498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.923470 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\") pod \"89aa5a36-166d-4e7c-a318-ae75b9d90085\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " Oct 30 22:40:47.923661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.923650 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\" (UniqueName: \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\") pod \"89aa5a36-166d-4e7c-a318-ae75b9d90085\" (UID: \"89aa5a36-166d-4e7c-a318-ae75b9d90085\") " Oct 30 22:40:47.923945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.923914 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume" pod "89aa5a36-166d-4e7c-a318-ae75b9d90085" (UID: "89aa5a36-166d-4e7c-a318-ae75b9d90085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:47.926347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.926320 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2" pod "89aa5a36-166d-4e7c-a318-ae75b9d90085" (UID: "89aa5a36-166d-4e7c-a318-ae75b9d90085"). InnerVolumeSpecName "kube-api-access-9g2d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:47.951213 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:47.951192 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:40:48.024509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.024483 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\") pod \"3a804b4b-4475-47fa-9181-acf2327826e3\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " Oct 30 22:40:48.024744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.024660 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\" (UniqueName: \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\") pod \"3a804b4b-4475-47fa-9181-acf2327826e3\" (UID: \"3a804b4b-4475-47fa-9181-acf2327826e3\") " Oct 30 22:40:48.024881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.024845 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume" pod "3a804b4b-4475-47fa-9181-acf2327826e3" (UID: "3a804b4b-4475-47fa-9181-acf2327826e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:48.025079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.025065 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2d2\" (UniqueName: \"kubernetes.io/projected/89aa5a36-166d-4e7c-a318-ae75b9d90085-kube-api-access-9g2d2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.025137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.025081 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89aa5a36-166d-4e7c-a318-ae75b9d90085-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.025137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.025091 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804b4b-4475-47fa-9181-acf2327826e3-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.026515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.026495 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs" pod "3a804b4b-4475-47fa-9181-acf2327826e3" (UID: "3a804b4b-4475-47fa-9181-acf2327826e3"). InnerVolumeSpecName "kube-api-access-kc6hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:48.125873 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.125805 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc6hs\" (UniqueName: \"kubernetes.io/projected/3a804b4b-4475-47fa-9181-acf2327826e3-kube-api-access-kc6hs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.221910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.221866 4292 generic.go:356] "Generic (PLEG): container finished" podID="3a804b4b-4475-47fa-9181-acf2327826e3" containerID="d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623" exitCode=0 Oct 30 22:40:48.222078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.221932 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" Oct 30 22:40:48.222078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.221940 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" event={"ID":"3a804b4b-4475-47fa-9181-acf2327826e3","Type":"ContainerDied","Data":"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623"} Oct 30 22:40:48.222078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.222055 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68" event={"ID":"3a804b4b-4475-47fa-9181-acf2327826e3","Type":"ContainerDied","Data":"fe04fd128c2d70b43c38931eeef69eb2d9133da69918ecbf81ca66813272cf3c"} Oct 30 22:40:48.222254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.222119 4292 scope.go:122] "RemoveContainer" containerID="d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623" Oct 30 22:40:48.223700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.223677 4292 generic.go:356] "Generic (PLEG): container finished" podID="89aa5a36-166d-4e7c-a318-ae75b9d90085" containerID="c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4" exitCode=0 Oct 30 22:40:48.223992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.223974 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" Oct 30 22:40:48.224086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.224008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" event={"ID":"89aa5a36-166d-4e7c-a318-ae75b9d90085","Type":"ContainerDied","Data":"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4"} Oct 30 22:40:48.224086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.224061 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n" event={"ID":"89aa5a36-166d-4e7c-a318-ae75b9d90085","Type":"ContainerDied","Data":"213a56ded25f57bb29f9b52cf193b8ca0efd4060546db4354abeac7e1ec891dc"} Oct 30 22:40:48.232487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.232455 4292 scope.go:122] "RemoveContainer" containerID="d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623" Oct 30 22:40:48.232788 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:40:48.232769 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623\": container with ID starting with d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623 not found: ID does not exist" containerID="d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623" Oct 30 22:40:48.232845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.232796 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623"} err="failed to get container status \"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623\": rpc error: code = NotFound desc = could not find container \"d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623\": container with ID starting with d35babf1922cdc9689276f6337de63d926c422c5611bb617071ac654f7a45623 not found: ID does not exist" Oct 30 22:40:48.232845 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.232814 4292 scope.go:122] "RemoveContainer" containerID="c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4" Oct 30 22:40:48.240867 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.240852 4292 scope.go:122] "RemoveContainer" containerID="c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4" Oct 30 22:40:48.241158 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:40:48.241120 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4\": container with ID starting with c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4 not found: ID does not exist" containerID="c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4" Oct 30 22:40:48.241225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.241166 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4"} err="failed to get container status \"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4\": rpc error: code = NotFound desc = could not find container \"c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4\": container with ID starting with c8f1c5599a5abcc80f11e2c7382df976602bd68ca2a3b94ddadbeb0667cb7cf4 not found: ID does not exist" Oct 30 22:40:48.391633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.391564 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:40:48.397879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.397858 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6c7d3f-64c79dd947-hms68"] Oct 30 22:40:48.512090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.512058 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:40:48.516883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:48.516860 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6c7d3f-f9b98f457-dsh6n"] Oct 30 22:40:50.144827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:50.144795 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a804b4b-4475-47fa-9181-acf2327826e3" path="/var/lib/kubelet/pods/3a804b4b-4475-47fa-9181-acf2327826e3/volumes" Oct 30 22:40:50.145210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:50.145155 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89aa5a36-166d-4e7c-a318-ae75b9d90085" path="/var/lib/kubelet/pods/89aa5a36-166d-4e7c-a318-ae75b9d90085/volumes" Oct 30 22:40:56.497003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:56.496961 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h"] Oct 30 22:40:56.547485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:56.547454 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6c7d3f-bl28h"] Oct 30 22:40:56.639622 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:40:56.639565 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:40:58.145876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:40:58.145845 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145f5063-c447-4196-b964-7ed5c0cf37fe" path="/var/lib/kubelet/pods/145f5063-c447-4196-b964-7ed5c0cf37fe/volumes" Oct 30 22:41:04.055813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:04.055781 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs"] Oct 30 22:41:04.106152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:04.106121 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777eca59-r6brs"] Oct 30 22:41:04.145664 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:04.145635 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00fe2c6-b237-4803-b744-cd291ca947c3" path="/var/lib/kubelet/pods/d00fe2c6-b237-4803-b744-cd291ca947c3/volumes" Oct 30 22:41:10.799467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:10.799434 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:41:10.849418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:10.849381 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:41:10.851615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:10.851589 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:41:21.910121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910090 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66"] Oct 30 22:41:21.910532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910509 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3a804b4b-4475-47fa-9181-acf2327826e3" containerName="payload-pre-processing" Oct 30 22:41:21.910532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910520 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a804b4b-4475-47fa-9181-acf2327826e3" containerName="payload-pre-processing" Oct 30 22:41:21.910635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910536 4292 cpu_manager.go:414] "containerMap: removing container" podUID="145f5063-c447-4196-b964-7ed5c0cf37fe" containerName="revoke-keys" Oct 30 22:41:21.910635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910546 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145f5063-c447-4196-b964-7ed5c0cf37fe" containerName="revoke-keys" Oct 30 22:41:21.910635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910585 4292 cpu_manager.go:414] "containerMap: removing container" podUID="89aa5a36-166d-4e7c-a318-ae75b9d90085" containerName="payload-processing" Oct 30 22:41:21.910635 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910594 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89aa5a36-166d-4e7c-a318-ae75b9d90085" containerName="payload-processing" Oct 30 22:41:21.910785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910666 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="145f5063-c447-4196-b964-7ed5c0cf37fe" containerName="revoke-keys" Oct 30 22:41:21.910785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910680 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="89aa5a36-166d-4e7c-a318-ae75b9d90085" containerName="payload-processing" Oct 30 22:41:21.910785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.910690 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a804b4b-4475-47fa-9181-acf2327826e3" containerName="payload-pre-processing" Oct 30 22:41:21.913742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.913721 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:21.933255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.933230 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh268\" (UniqueName: \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:21.933370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.933355 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:21.933424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.933383 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:21.974535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:21.974508 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66"] Oct 30 22:41:22.022097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.022072 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:22.022097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.022080 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:41:22.022250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.022107 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:41:22.022250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.022187 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:22.034627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.034608 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh268\" (UniqueName: \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.034708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.034679 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.034708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.034703 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.035347 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.035326 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.037265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.037245 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.097643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.097620 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh268\" (UniqueName: \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\") pod \"maas-api-revoke-sub-f3c24d18-frc66\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.227408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.227353 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:22.601972 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:41:22.601939 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9599455_7d55_45fb_85f2_e0a303539ff1.slice/crio-68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4 WatchSource:0}: Error finding container 68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4: Status 404 returned error can't find the container with id 68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4 Oct 30 22:41:22.602165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:22.602140 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66"] Oct 30 22:41:23.428051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:23.428014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" event={"ID":"f9599455-7d55-45fb-85f2-e0a303539ff1","Type":"ContainerStarted","Data":"a220187312e683f7334d12f37947edbee5098e548acb8d728a7723fed3a33566"} Oct 30 22:41:23.428568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:23.428070 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" event={"ID":"f9599455-7d55-45fb-85f2-e0a303539ff1","Type":"ContainerStarted","Data":"68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4"} Oct 30 22:41:23.548187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:23.548140 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" podStartSLOduration=2.548126952 podStartE2EDuration="2.548126952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:23.546778243 +0000 UTC m=+2811.515597808" watchObservedRunningTime="2026-10-30 22:41:23.548126952 +0000 UTC m=+2811.516946510" Oct 30 22:41:24.435060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:24.435023 4292 generic.go:356] "Generic (PLEG): container finished" podID="f9599455-7d55-45fb-85f2-e0a303539ff1" containerID="a220187312e683f7334d12f37947edbee5098e548acb8d728a7723fed3a33566" exitCode=0 Oct 30 22:41:24.435060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:24.435062 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" event={"ID":"f9599455-7d55-45fb-85f2-e0a303539ff1","Type":"ContainerDied","Data":"a220187312e683f7334d12f37947edbee5098e548acb8d728a7723fed3a33566"} Oct 30 22:41:25.580468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.580443 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:25.669666 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.669641 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\") pod \"f9599455-7d55-45fb-85f2-e0a303539ff1\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " Oct 30 22:41:25.669808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.669704 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\") pod \"f9599455-7d55-45fb-85f2-e0a303539ff1\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " Oct 30 22:41:25.669875 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.669803 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\" (UniqueName: \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\") pod \"f9599455-7d55-45fb-85f2-e0a303539ff1\" (UID: \"f9599455-7d55-45fb-85f2-e0a303539ff1\") " Oct 30 22:41:25.670028 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.670008 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca" pod "f9599455-7d55-45fb-85f2-e0a303539ff1" (UID: "f9599455-7d55-45fb-85f2-e0a303539ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:25.670188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.670172 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:25.671862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.671839 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca" pod "f9599455-7d55-45fb-85f2-e0a303539ff1" (UID: "f9599455-7d55-45fb-85f2-e0a303539ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:25.671949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.671872 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268" pod "f9599455-7d55-45fb-85f2-e0a303539ff1" (UID: "f9599455-7d55-45fb-85f2-e0a303539ff1"). InnerVolumeSpecName "kube-api-access-rh268". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:25.771570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.771504 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh268\" (UniqueName: \"kubernetes.io/projected/f9599455-7d55-45fb-85f2-e0a303539ff1-kube-api-access-rh268\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:25.771570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.771533 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9599455-7d55-45fb-85f2-e0a303539ff1-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:25.820636 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.820560 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" containerID="cri-o://1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd" gracePeriod=45 Oct 30 22:41:25.869149 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.869117 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" containerID="cri-o://74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51" gracePeriod=45 Oct 30 22:41:25.870981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:25.870946 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" containerID="cri-o://0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c" gracePeriod=45 Oct 30 22:41:26.124457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.124427 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:41:26.138514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.138490 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:41:26.148514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.148485 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:41:26.175561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.175701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175590 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.175701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175614 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.175701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175688 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175710 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175744 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\" (UniqueName: \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175747 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175855 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.175888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175874 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175897 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\" (UniqueName: \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175923 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175947 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\") pod \"439bff93-844c-41b5-ba0e-705bbf8db3a5\" (UID: \"439bff93-844c-41b5-ba0e-705bbf8db3a5\") " Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175988 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.175994 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\" (UniqueName: \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\") pod \"132667f4-0f29-44a0-8951-4e7ac378320c\" (UID: \"132667f4-0f29-44a0-8951-4e7ac378320c\") " Oct 30 22:41:26.176217 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176118 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176311 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176376 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176504 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176527 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176542 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.176555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176555 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.176913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176584 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.176913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.176599 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.178348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178320 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.178451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178340 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "kube-api-access-gm6jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:26.178451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178395 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:26.178451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178435 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "kube-api-access-mvvfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:26.178601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178436 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm" pod "439bff93-844c-41b5-ba0e-705bbf8db3a5" (UID: "439bff93-844c-41b5-ba0e-705bbf8db3a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.178795 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.178780 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs" pod "132667f4-0f29-44a0-8951-4e7ac378320c" (UID: "132667f4-0f29-44a0-8951-4e7ac378320c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:26.277866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.277832 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\" (UniqueName: \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.277875 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.277951 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278060 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278108 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278165 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\") pod \"7b33a316-fb2d-42e7-869e-fefc2700863f\" (UID: \"7b33a316-fb2d-42e7-869e-fefc2700863f\") " Oct 30 22:41:26.278428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278400 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.278615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278586 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278611 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439bff93-844c-41b5-ba0e-705bbf8db3a5-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278627 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132667f4-0f29-44a0-8951-4e7ac378320c-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278642 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvfx\" (UniqueName: \"kubernetes.io/projected/439bff93-844c-41b5-ba0e-705bbf8db3a5-kube-api-access-mvvfx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278659 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132667f4-0f29-44a0-8951-4e7ac378320c-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278673 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6jx\" (UniqueName: \"kubernetes.io/projected/132667f4-0f29-44a0-8951-4e7ac378320c-kube-api-access-gm6jx\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.278774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278687 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439bff93-844c-41b5-ba0e-705bbf8db3a5-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.279074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278838 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.279074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.278856 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.280811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.280743 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "kube-api-access-xbcnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:26.280981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.280893 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:26.281890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.281864 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm" pod "7b33a316-fb2d-42e7-869e-fefc2700863f" (UID: "7b33a316-fb2d-42e7-869e-fefc2700863f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:26.379468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.379440 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b33a316-fb2d-42e7-869e-fefc2700863f-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.379468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.379464 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.379681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.379475 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.379681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.379485 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcnt\" (UniqueName: \"kubernetes.io/projected/7b33a316-fb2d-42e7-869e-fefc2700863f-kube-api-access-xbcnt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.379681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.379495 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b33a316-fb2d-42e7-869e-fefc2700863f-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:26.448888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.448856 4292 generic.go:356] "Generic (PLEG): container finished" podID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerID="0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c" exitCode=0 Oct 30 22:41:26.449070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.448901 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" Oct 30 22:41:26.449070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.448948 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" event={"ID":"439bff93-844c-41b5-ba0e-705bbf8db3a5","Type":"ContainerDied","Data":"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c"} Oct 30 22:41:26.449070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.449010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm" event={"ID":"439bff93-844c-41b5-ba0e-705bbf8db3a5","Type":"ContainerDied","Data":"901a093d6e44385cbeb3d2070197a6039a922a4b151117bb7967dca8a802e56a"} Oct 30 22:41:26.449070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.449029 4292 scope.go:122] "RemoveContainer" containerID="0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c" Oct 30 22:41:26.451006 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.450977 4292 generic.go:356] "Generic (PLEG): container finished" podID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerID="74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51" exitCode=0 Oct 30 22:41:26.451147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.451010 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" Oct 30 22:41:26.451147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.451052 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" event={"ID":"7b33a316-fb2d-42e7-869e-fefc2700863f","Type":"ContainerDied","Data":"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51"} Oct 30 22:41:26.451147 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.451088 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6" event={"ID":"7b33a316-fb2d-42e7-869e-fefc2700863f","Type":"ContainerDied","Data":"60fec557aa16900316d16042a709f01ec52b01494720123e5f4002cf1780b3b6"} Oct 30 22:41:26.454837 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.454812 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" Oct 30 22:41:26.454969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.454873 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66" event={"ID":"f9599455-7d55-45fb-85f2-e0a303539ff1","Type":"ContainerDied","Data":"68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4"} Oct 30 22:41:26.454969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.454914 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68424a8be2f8043b7698f7e0231e04d6a46bbf15efdcca0b713bce5d592c64f4" Oct 30 22:41:26.456627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.456568 4292 generic.go:356] "Generic (PLEG): container finished" podID="132667f4-0f29-44a0-8951-4e7ac378320c" containerID="1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd" exitCode=0 Oct 30 22:41:26.456627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.456617 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" Oct 30 22:41:26.456861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.456683 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" event={"ID":"132667f4-0f29-44a0-8951-4e7ac378320c","Type":"ContainerDied","Data":"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd"} Oct 30 22:41:26.456861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.456711 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg" event={"ID":"132667f4-0f29-44a0-8951-4e7ac378320c","Type":"ContainerDied","Data":"d83c0716fc61c0aa6b2c113e5161133071adf8c3387d5fc3c6903aede7adc435"} Oct 30 22:41:26.461302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.461284 4292 scope.go:122] "RemoveContainer" containerID="0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c" Oct 30 22:41:26.461646 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:26.461615 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c\": container with ID starting with 0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c not found: ID does not exist" containerID="0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c" Oct 30 22:41:26.461760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.461655 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c"} err="failed to get container status \"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c\": rpc error: code = NotFound desc = could not find container \"0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c\": container with ID starting with 0dd5ba5ed52d85e4b4e3dac124852fb9f5e438165ea76f51c588b2608412a31c not found: ID does not exist" Oct 30 22:41:26.461760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.461680 4292 scope.go:122] "RemoveContainer" containerID="74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51" Oct 30 22:41:26.470406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.470386 4292 scope.go:122] "RemoveContainer" containerID="74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51" Oct 30 22:41:26.470781 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:26.470761 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51\": container with ID starting with 74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51 not found: ID does not exist" containerID="74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51" Oct 30 22:41:26.470872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.470789 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51"} err="failed to get container status \"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51\": rpc error: code = NotFound desc = could not find container \"74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51\": container with ID starting with 74c09f86d0804b2a96ce12ec2d606666fcbf2e184cc07606b67df50b086dbe51 not found: ID does not exist" Oct 30 22:41:26.470872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.470813 4292 scope.go:122] "RemoveContainer" containerID="1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd" Oct 30 22:41:26.479939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.479921 4292 scope.go:122] "RemoveContainer" containerID="1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd" Oct 30 22:41:26.480199 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:26.480179 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd\": container with ID starting with 1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd not found: ID does not exist" containerID="1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd" Oct 30 22:41:26.480265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.480208 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd"} err="failed to get container status \"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd\": rpc error: code = NotFound desc = could not find container \"1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd\": container with ID starting with 1d10a293bb6a92548b5da515aab4d3ab4011c0ceb35392f3d93c3b6aa75abccd not found: ID does not exist" Oct 30 22:41:26.639145 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:26.639071 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:41:26.661726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.661697 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:41:26.675488 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.669917 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/distinct-w6-c76d9e-kserve-54699674bf-7nkpg"] Oct 30 22:41:26.789752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.789723 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:41:26.796497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.796476 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/simulator-w6-c76d9e-kserve-7b5b7d6fb7-wczvm"] Oct 30 22:41:26.966370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.966297 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:41:26.974483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:26.974457 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c76d9e/premium-w6-c76d9e-kserve-6f5cf4c594-vbpx6"] Oct 30 22:41:28.145351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:28.145320 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" path="/var/lib/kubelet/pods/132667f4-0f29-44a0-8951-4e7ac378320c/volumes" Oct 30 22:41:28.145737 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:28.145728 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" path="/var/lib/kubelet/pods/439bff93-844c-41b5-ba0e-705bbf8db3a5/volumes" Oct 30 22:41:28.146097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:28.146083 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" path="/var/lib/kubelet/pods/7b33a316-fb2d-42e7-869e-fefc2700863f/volumes" Oct 30 22:41:34.056434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.056396 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl"] Oct 30 22:41:34.106746 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.106716 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx"] Oct 30 22:41:34.108678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.108657 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6b24e1-5n8wl"] Oct 30 22:41:34.110324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.110300 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdbfac0-jrxtx"] Oct 30 22:41:34.145995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.145960 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680e2501-67fe-4568-bdf8-041dc19b410c" path="/var/lib/kubelet/pods/680e2501-67fe-4568-bdf8-041dc19b410c/volumes" Oct 30 22:41:34.146391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:34.146374 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83569574-1464-4a63-8c70-8b328e5be71f" path="/var/lib/kubelet/pods/83569574-1464-4a63-8c70-8b328e5be71f/volumes" Oct 30 22:41:37.877046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:37.877015 4292 scope.go:122] "RemoveContainer" containerID="2838ddffe2194a77a9b990e54bdff176c84045603d1bcae24cafdb46cc5287d2" Oct 30 22:41:37.888398 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:37.888374 4292 scope.go:122] "RemoveContainer" containerID="e728e8a591eb78571a1b85dafd193a8a0166ad8891890ad5e768343dc979352d" Oct 30 22:41:37.896714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:37.896692 4292 scope.go:122] "RemoveContainer" containerID="18cb15e231156704c2c70686f8cbba4f81d58e5713234718cd30528861b81574" Oct 30 22:41:47.595218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.595127 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:41:47.595704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.595354 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" containerID="cri-o://ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627" gracePeriod=30 Oct 30 22:41:47.645343 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.645313 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:41:47.645562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.645534 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" containerID="cri-o://f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4" gracePeriod=30 Oct 30 22:41:47.647327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.647304 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:41:47.647505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.647471 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" containerID="cri-o://9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2" gracePeriod=30 Oct 30 22:41:47.696816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.696782 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:41:47.697045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:47.697020 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" containerID="cri-o://051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f" gracePeriod=30 Oct 30 22:41:48.640805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:48.640766 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jdf2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:48.641184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:48.640847 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:49.060503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:49.060467 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445"] Oct 30 22:41:49.063117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:49.063094 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438de563-dh445"] Oct 30 22:41:49.804857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:49.804826 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:41:49.805628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:49.805594 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" containerID="cri-o://b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a" gracePeriod=30 Oct 30 22:41:50.054485 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:50.054446 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx"] Oct 30 22:41:50.059433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:50.059362 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beea9ec0-7xkzx"] Oct 30 22:41:50.145329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:50.145299 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0" path="/var/lib/kubelet/pods/9fa64f4f-4fdb-4d22-a0d9-ac1b57a451c0/volumes" Oct 30 22:41:50.145803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:50.145785 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dd4e74-a9f0-456a-9fce-b0b207d6bc93" path="/var/lib/kubelet/pods/e9dd4e74-a9f0-456a-9fce-b0b207d6bc93/volumes" Oct 30 22:41:52.932773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:52.932745 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:41:52.973863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:52.973838 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:41:52.981852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:52.981833 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:41:52.984829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:52.984812 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:41:53.014724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014703 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.014833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014751 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.014833 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014782 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.014926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014854 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.014926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014882 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.014926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014916 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.015127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014945 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.015127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.014946 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.015127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015025 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.015127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015054 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\") pod \"d8c62adc-bb80-4507-9d92-744a3fa751ab\" (UID: \"d8c62adc-bb80-4507-9d92-744a3fa751ab\") " Oct 30 22:41:53.015318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015203 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.015417 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015394 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.015507 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015446 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:53.015584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015510 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.015584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015529 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.015693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015600 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.015693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.015627 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.017018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.016999 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.017139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.017128 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.017437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.017413 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:53.017483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.017417 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf" pod "d8c62adc-bb80-4507-9d92-744a3fa751ab" (UID: "d8c62adc-bb80-4507-9d92-744a3fa751ab"). InnerVolumeSpecName "kube-api-access-krxmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.116172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116142 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116180 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116203 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116231 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116259 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116293 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.116342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116325 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116360 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116407 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116460 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116489 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116524 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116561 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116614 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116668 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.116728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116708 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116741 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116768 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116802 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116837 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116882 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116925 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116956 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116987 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117030 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\") pod \"13eb4111-604b-4ff3-86a1-81d42059a42d\" (UID: \"13eb4111-604b-4ff3-86a1-81d42059a42d\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117062 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\") pod \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\" (UID: \"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e\") " Oct 30 22:41:53.117227 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117092 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\") pod \"4b538b4d-25a3-4ce6-905d-57411194abcd\" (UID: \"4b538b4d-25a3-4ce6-905d-57411194abcd\") " Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117474 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxmf\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-kube-api-access-krxmf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117497 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117511 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62adc-bb80-4507-9d92-744a3fa751ab-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117525 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117539 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.117871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117555 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62adc-bb80-4507-9d92-744a3fa751ab-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.118202 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116459 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.116990 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117206 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117430 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117617 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117811 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.117862 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118013 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118095 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118263 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.118866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118404 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:53.118866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118668 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:53.118866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.118688 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.119162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.119130 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.119462 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.119435 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:53.120918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.120895 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "kube-api-access-dg4k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.121305 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.121283 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.121897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.121877 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:53.121997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.121909 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.121997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.121943 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "kube-api-access-mbwpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.122143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.122010 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "kube-api-access-fhsfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.122335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.122288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.122335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.122301 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:53.122489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.122331 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token" pod "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" (UID: "9d28cc4d-575c-41d1-9587-3ecd6acc1a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.122489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.122345 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token" pod "4b538b4d-25a3-4ce6-905d-57411194abcd" (UID: "4b538b4d-25a3-4ce6-905d-57411194abcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:53.123376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.123356 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo" pod "13eb4111-604b-4ff3-86a1-81d42059a42d" (UID: "13eb4111-604b-4ff3-86a1-81d42059a42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218426 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b538b4d-25a3-4ce6-905d-57411194abcd-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218452 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218463 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218473 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218482 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218490 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218499 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218507 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218516 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218524 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218531 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhsfc\" (UniqueName: \"kubernetes.io/projected/13eb4111-604b-4ff3-86a1-81d42059a42d-kube-api-access-fhsfc\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218541 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218550 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218559 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218568 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13eb4111-604b-4ff3-86a1-81d42059a42d-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218599 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218608 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218615 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218625 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwpv\" (UniqueName: \"kubernetes.io/projected/4b538b4d-25a3-4ce6-905d-57411194abcd-kube-api-access-mbwpv\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218632 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218641 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218648 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4k9\" (UniqueName: \"kubernetes.io/projected/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-kube-api-access-dg4k9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218657 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218665 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218674 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218682 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.218813 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.218690 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13eb4111-604b-4ff3-86a1-81d42059a42d-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:53.604053 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.604016 4292 generic.go:356] "Generic (PLEG): container finished" podID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerID="f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4" exitCode=0 Oct 30 22:41:53.604233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.604067 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" Oct 30 22:41:53.604233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.604101 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" event={"ID":"4b538b4d-25a3-4ce6-905d-57411194abcd","Type":"ContainerDied","Data":"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4"} Oct 30 22:41:53.604233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.604155 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g" event={"ID":"4b538b4d-25a3-4ce6-905d-57411194abcd","Type":"ContainerDied","Data":"14fb8f7c9cfa3e6e01f865b981dcf736e8cdad7041d52f73ac1c4a5dc823ea9d"} Oct 30 22:41:53.604233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.604188 4292 scope.go:122] "RemoveContainer" containerID="f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4" Oct 30 22:41:53.605553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.605525 4292 generic.go:356] "Generic (PLEG): container finished" podID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerID="ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627" exitCode=0 Oct 30 22:41:53.605553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.605553 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" event={"ID":"d8c62adc-bb80-4507-9d92-744a3fa751ab","Type":"ContainerDied","Data":"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627"} Oct 30 22:41:53.605774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.605602 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" Oct 30 22:41:53.605774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.605621 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp" event={"ID":"d8c62adc-bb80-4507-9d92-744a3fa751ab","Type":"ContainerDied","Data":"b38a12143b9c57b3cc4e0223a472d866d9d10a5e3d43760ba6f362902201db05"} Oct 30 22:41:53.607104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.607082 4292 generic.go:356] "Generic (PLEG): container finished" podID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerID="051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f" exitCode=0 Oct 30 22:41:53.607218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.607115 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" event={"ID":"13eb4111-604b-4ff3-86a1-81d42059a42d","Type":"ContainerDied","Data":"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f"} Oct 30 22:41:53.607218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.607139 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" Oct 30 22:41:53.607218 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.607175 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" event={"ID":"13eb4111-604b-4ff3-86a1-81d42059a42d","Type":"ContainerDied","Data":"f352017f077961d2f8ce1b8b07057ade2fa4c664c46ac5157fdb9c2c11f5f004"} Oct 30 22:41:53.608733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.608709 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerID="9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2" exitCode=0 Oct 30 22:41:53.608850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.608747 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" event={"ID":"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e","Type":"ContainerDied","Data":"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2"} Oct 30 22:41:53.608850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.608757 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" Oct 30 22:41:53.608850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.608785 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv" event={"ID":"9d28cc4d-575c-41d1-9587-3ecd6acc1a5e","Type":"ContainerDied","Data":"ccb625b37a548469e3fbe2230a586cc214f7cb1b7b6f5f07cb9cfd22f1e42d59"} Oct 30 22:41:53.621712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.621682 4292 scope.go:122] "RemoveContainer" containerID="f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4" Oct 30 22:41:53.624256 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:53.624127 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4\": container with ID starting with f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4 not found: ID does not exist" containerID="f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4" Oct 30 22:41:53.624256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.624161 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4"} err="failed to get container status \"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4\": rpc error: code = NotFound desc = could not find container \"f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4\": container with ID starting with f43ab484b027de6fb4d2a7d94be9e6f5a95c1462d0751b3ba84dcc84be476ce4 not found: ID does not exist" Oct 30 22:41:53.624256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.624183 4292 scope.go:122] "RemoveContainer" containerID="ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627" Oct 30 22:41:53.647518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.647454 4292 scope.go:122] "RemoveContainer" containerID="ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627" Oct 30 22:41:53.647907 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:53.647823 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627\": container with ID starting with ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627 not found: ID does not exist" containerID="ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627" Oct 30 22:41:53.647907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.647854 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627"} err="failed to get container status \"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627\": rpc error: code = NotFound desc = could not find container \"ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627\": container with ID starting with ed5ec96bdae8589cca66252d7b0ca8c2ecdbc79e5aa3fde95a2fcda3ac00f627 not found: ID does not exist" Oct 30 22:41:53.647907 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.647877 4292 scope.go:122] "RemoveContainer" containerID="051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f" Oct 30 22:41:53.655990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.655974 4292 scope.go:122] "RemoveContainer" containerID="051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f" Oct 30 22:41:53.656231 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:53.656215 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f\": container with ID starting with 051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f not found: ID does not exist" containerID="051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f" Oct 30 22:41:53.656276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.656237 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f"} err="failed to get container status \"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f\": rpc error: code = NotFound desc = could not find container \"051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f\": container with ID starting with 051b9e7006ab3f47b675732c933d1c04d2cfa854c8bcbaf57f383f5415451f1f not found: ID does not exist" Oct 30 22:41:53.656276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.656251 4292 scope.go:122] "RemoveContainer" containerID="9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2" Oct 30 22:41:53.664359 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.664339 4292 scope.go:122] "RemoveContainer" containerID="9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2" Oct 30 22:41:53.664661 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:53.664630 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2\": container with ID starting with 9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2 not found: ID does not exist" containerID="9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2" Oct 30 22:41:53.664743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.664658 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2"} err="failed to get container status \"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2\": rpc error: code = NotFound desc = could not find container \"9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2\": container with ID starting with 9b71657d1641d7fea4c8f838176ea473a391030d6f244e0cf0a6a315b2ec57f2 not found: ID does not exist" Oct 30 22:41:53.776820 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.776792 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:41:53.789027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.789002 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-slvnp"] Oct 30 22:41:53.910563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.910505 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:41:53.916386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.916365 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5xmvv"] Oct 30 22:41:53.956207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.956178 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5k6lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:41:53.956537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:53.956225 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:41:54.029738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.029710 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:41:54.036290 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.036265 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jdf2g"] Oct 30 22:41:54.145018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.144990 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" path="/var/lib/kubelet/pods/4b538b4d-25a3-4ce6-905d-57411194abcd/volumes" Oct 30 22:41:54.145427 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.145414 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" path="/var/lib/kubelet/pods/9d28cc4d-575c-41d1-9587-3ecd6acc1a5e/volumes" Oct 30 22:41:54.145827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.145815 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" path="/var/lib/kubelet/pods/d8c62adc-bb80-4507-9d92-744a3fa751ab/volumes" Oct 30 22:41:54.148824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.148803 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:41:54.154519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:54.154496 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5k6lk"] Oct 30 22:41:55.052295 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.052273 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:41:55.137195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137140 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137195 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137186 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137234 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137296 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137330 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137350 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:55.137487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137396 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137430 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137464 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137508 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\") pod \"e6618093-a44d-42a7-b805-2a9282c9d057\" (UID: \"e6618093-a44d-42a7-b805-2a9282c9d057\") " Oct 30 22:41:55.137749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137707 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:55.137749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137735 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:55.137749 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137734 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:55.137980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137916 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:55.137980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137941 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.137980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137962 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.137980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137978 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.138209 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.137992 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6618093-a44d-42a7-b805-2a9282c9d057-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.139524 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.139475 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:55.139683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.139551 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "kube-api-access-2rgzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:55.139683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.139558 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:55.139683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.139667 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo" pod "e6618093-a44d-42a7-b805-2a9282c9d057" (UID: "e6618093-a44d-42a7-b805-2a9282c9d057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:55.239165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.239141 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.239165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.239162 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.239332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.239173 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgzr\" (UniqueName: \"kubernetes.io/projected/e6618093-a44d-42a7-b805-2a9282c9d057-kube-api-access-2rgzr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.239332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.239183 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6618093-a44d-42a7-b805-2a9282c9d057-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.239332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.239193 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6618093-a44d-42a7-b805-2a9282c9d057-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:55.622919 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.622880 4292 generic.go:356] "Generic (PLEG): container finished" podID="e6618093-a44d-42a7-b805-2a9282c9d057" containerID="b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a" exitCode=0 Oct 30 22:41:55.623098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.622938 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" Oct 30 22:41:55.623098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.622963 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" event={"ID":"e6618093-a44d-42a7-b805-2a9282c9d057","Type":"ContainerDied","Data":"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a"} Oct 30 22:41:55.623098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.623023 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w" event={"ID":"e6618093-a44d-42a7-b805-2a9282c9d057","Type":"ContainerDied","Data":"64aa3095588279eb6dd5e3b45ae33ff81e9365b2d284feb7eb74d0e30a6c5c03"} Oct 30 22:41:55.623098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.623048 4292 scope.go:122] "RemoveContainer" containerID="b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a" Oct 30 22:41:55.632980 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.632961 4292 scope.go:122] "RemoveContainer" containerID="b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a" Oct 30 22:41:55.633252 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:55.633231 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a\": container with ID starting with b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a not found: ID does not exist" containerID="b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a" Oct 30 22:41:55.633308 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.633261 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a"} err="failed to get container status \"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a\": rpc error: code = NotFound desc = could not find container \"b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a\": container with ID starting with b80599a02956bcd2eb5a7e7204f301077f35a47c01335c0195ae0dbcf242618a not found: ID does not exist" Oct 30 22:41:55.792624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.792591 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:41:55.843454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:55.843413 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c76d9e-openshift-default-5dddb76f45-5lw2w"] Oct 30 22:41:56.145197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:56.145159 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" path="/var/lib/kubelet/pods/13eb4111-604b-4ff3-86a1-81d42059a42d/volumes" Oct 30 22:41:56.145744 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:41:56.145612 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" path="/var/lib/kubelet/pods/e6618093-a44d-42a7-b805-2a9282c9d057/volumes" Oct 30 22:41:56.639926 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:41:56.639889 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:42:17.879111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879075 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw"] Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879634 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f9599455-7d55-45fb-85f2-e0a303539ff1" containerName="revoke-keys" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879656 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9599455-7d55-45fb-85f2-e0a303539ff1" containerName="revoke-keys" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879670 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879678 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879702 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879711 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879734 4292 cpu_manager.go:414] "containerMap: removing container" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879743 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879755 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879762 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879771 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" Oct 30 22:42:17.879785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879778 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879794 4292 cpu_manager.go:414] "containerMap: removing container" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879801 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879823 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879831 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879850 4292 cpu_manager.go:414] "containerMap: removing container" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879857 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879944 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="439bff93-844c-41b5-ba0e-705bbf8db3a5" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879960 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d28cc4d-575c-41d1-9587-3ecd6acc1a5e" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879973 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="132667f4-0f29-44a0-8951-4e7ac378320c" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879986 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9599455-7d55-45fb-85f2-e0a303539ff1" containerName="revoke-keys" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.879993 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b33a316-fb2d-42e7-869e-fefc2700863f" containerName="main" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.880002 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b538b4d-25a3-4ce6-905d-57411194abcd" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.880014 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="13eb4111-604b-4ff3-86a1-81d42059a42d" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.880023 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6618093-a44d-42a7-b805-2a9282c9d057" containerName="istio-proxy" Oct 30 22:42:17.880441 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.880032 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c62adc-bb80-4507-9d92-744a3fa751ab" containerName="istio-proxy" Oct 30 22:42:17.882900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.882880 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927363 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927409 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927435 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927482 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927595 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927628 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwplz\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-kube-api-access-hwplz\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927731 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927771 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.927896 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.927836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:17.929452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.929432 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj"] Oct 30 22:42:17.932738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.932725 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:17.983124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:17.983103 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw"] Oct 30 22:42:18.028344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028312 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.028491 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028370 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.028555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028485 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.028555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028525 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.028699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028558 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.028785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028711 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.028785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028759 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.029003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028791 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029003 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.028817 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qnf\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-kube-api-access-49qnf\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.029219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029192 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029237 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029243 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.029354 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029332 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029356 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029378 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029405 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029436 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029501 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwplz\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-kube-api-access-hwplz\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.029871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029531 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.030283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.029915 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.030283 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.030152 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.032403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.032377 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.032790 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.032765 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 22:42:18.032963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.032939 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.034340 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.034317 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj"] Oct 30 22:42:18.034791 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.034772 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9qrfj\"" Oct 30 22:42:18.040420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.040396 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.092184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.092162 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.092570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.092551 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwplz\" (UniqueName: \"kubernetes.io/projected/71ff446a-b780-4219-aef2-f8e8c4bd3cbf-kube-api-access-hwplz\") pod \"maas-default-gateway-openshift-default-76597d6855-rb4gw\" (UID: \"71ff446a-b780-4219-aef2-f8e8c4bd3cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.129975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.129912 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qnf\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-kube-api-access-49qnf\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.129980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130000 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130019 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130055 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130099 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130091 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130135 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130163 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130371 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130210 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130429 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130487 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130552 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.130701 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.130689 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.131029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.131006 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.132177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.132152 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.132460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.132444 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.193891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.193871 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.194780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.194757 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:18.207038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.207013 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qnf\" (UniqueName: \"kubernetes.io/projected/db3c85c0-bdfa-44de-9b36-6bde1bd6bba4-kube-api-access-49qnf\") pod \"maas-default-gateway-openshift-default-76597d6855-z8pdj\" (UID: \"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.244048 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.244017 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:18.595828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.591749 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7"] Oct 30 22:42:18.598243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.598218 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.623517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.623489 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw"] Oct 30 22:42:18.634971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.634945 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.635092 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.635071 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.635141 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.635123 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kzn\" (UniqueName: \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.654835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.654812 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:42:18.655527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.655501 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:42:18.656180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.656159 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:42:18.656722 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:42:18.656700 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ff446a_b780_4219_aef2_f8e8c4bd3cbf.slice/crio-6a870add47d9ab0670769f6fd87b676c2dce6a9bd2563b45ed706a4a5e3704e1 WatchSource:0}: Error finding container 6a870add47d9ab0670769f6fd87b676c2dce6a9bd2563b45ed706a4a5e3704e1: Status 404 returned error can't find the container with id 6a870add47d9ab0670769f6fd87b676c2dce6a9bd2563b45ed706a4a5e3704e1 Oct 30 22:42:18.658164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.658141 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7"] Oct 30 22:42:18.660216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.660185 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.660362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.660260 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.660362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.660307 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.677043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.677024 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj"] Oct 30 22:42:18.679757 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:42:18.679732 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c85c0_bdfa_44de_9b36_6bde1bd6bba4.slice/crio-8b55f1da7b4992ae71d800c6ea96280e2982fb40c937691dfc873ad50bddb84c WatchSource:0}: Error finding container 8b55f1da7b4992ae71d800c6ea96280e2982fb40c937691dfc873ad50bddb84c: Status 404 returned error can't find the container with id 8b55f1da7b4992ae71d800c6ea96280e2982fb40c937691dfc873ad50bddb84c Oct 30 22:42:18.682394 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.682360 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.682510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.682438 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.682510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.682483 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:18.736145 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.736109 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.736277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.736187 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kzn\" (UniqueName: \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.736277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.736217 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.736997 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.736956 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.738284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.738257 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.749220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.749185 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" event={"ID":"71ff446a-b780-4219-aef2-f8e8c4bd3cbf","Type":"ContainerStarted","Data":"cb67bea2f58b65152be1695ab48802bc1ae35573575a1d020b82ab071992b02a"} Oct 30 22:42:18.749337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.749253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" event={"ID":"71ff446a-b780-4219-aef2-f8e8c4bd3cbf","Type":"ContainerStarted","Data":"6a870add47d9ab0670769f6fd87b676c2dce6a9bd2563b45ed706a4a5e3704e1"} Oct 30 22:42:18.752302 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.752273 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" event={"ID":"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4","Type":"ContainerStarted","Data":"751d97f2d13218d469fb1eecbdccae1590c669b6980512c50b1d3ee7fa7e5c7d"} Oct 30 22:42:18.752399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.752328 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" event={"ID":"db3c85c0-bdfa-44de-9b36-6bde1bd6bba4","Type":"ContainerStarted","Data":"8b55f1da7b4992ae71d800c6ea96280e2982fb40c937691dfc873ad50bddb84c"} Oct 30 22:42:18.757743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.757712 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:42:18.800626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.800558 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kzn\" (UniqueName: \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\") pod \"maas-api-revoke-sub-85237e45-fxzx7\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:18.875922 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.875833 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podStartSLOduration=1.875819173 podStartE2EDuration="1.875819173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:18.873615539 +0000 UTC m=+2866.842435106" watchObservedRunningTime="2026-10-30 22:42:18.875819173 +0000 UTC m=+2866.844638734" Oct 30 22:42:18.912047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:18.912016 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:19.207677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.195958 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:19.207677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.201688 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:19.207677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.201739 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:19.245384 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.245343 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:19.246706 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.246677 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:19.246871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.246734 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:19.301595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.293810 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podStartSLOduration=2.293788461 podStartE2EDuration="2.293788461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:19.003690806 +0000 UTC m=+2866.972510369" watchObservedRunningTime="2026-10-30 22:42:19.293788461 +0000 UTC m=+2867.262608028" Oct 30 22:42:19.319600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.314669 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7"] Oct 30 22:42:19.322643 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:42:19.321810 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72aa6e74_5fcd_478f_a36f_871a0672a615.slice/crio-242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a WatchSource:0}: Error finding container 242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a: Status 404 returned error can't find the container with id 242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a Oct 30 22:42:19.759662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.759624 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" event={"ID":"72aa6e74-5fcd-478f-a36f-871a0672a615","Type":"ContainerStarted","Data":"ddc2db18307700d8ed47d1ef74eda8aa3fbbfe5a7564d3edc017c14b1c3a9b83"} Oct 30 22:42:19.759832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:19.759680 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" event={"ID":"72aa6e74-5fcd-478f-a36f-871a0672a615","Type":"ContainerStarted","Data":"242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a"} Oct 30 22:42:20.204598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.195501 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:20.204598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.195596 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:20.245508 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.245470 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:20.245710 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.245526 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:20.767432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.767399 4292 generic.go:356] "Generic (PLEG): container finished" podID="72aa6e74-5fcd-478f-a36f-871a0672a615" containerID="ddc2db18307700d8ed47d1ef74eda8aa3fbbfe5a7564d3edc017c14b1c3a9b83" exitCode=0 Oct 30 22:42:20.767671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:20.767464 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" event={"ID":"72aa6e74-5fcd-478f-a36f-871a0672a615","Type":"ContainerDied","Data":"ddc2db18307700d8ed47d1ef74eda8aa3fbbfe5a7564d3edc017c14b1c3a9b83"} Oct 30 22:42:21.201050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:21.201009 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:21.201338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:21.201308 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:21.245083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:21.245045 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:21.245452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:21.245093 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:22.041259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.041202 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:22.072143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.072118 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\") pod \"72aa6e74-5fcd-478f-a36f-871a0672a615\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " Oct 30 22:42:22.072350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.072337 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\" (UniqueName: \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\") pod \"72aa6e74-5fcd-478f-a36f-871a0672a615\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " Oct 30 22:42:22.072464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.072450 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\") pod \"72aa6e74-5fcd-478f-a36f-871a0672a615\" (UID: \"72aa6e74-5fcd-478f-a36f-871a0672a615\") " Oct 30 22:42:22.072627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.072457 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca" pod "72aa6e74-5fcd-478f-a36f-871a0672a615" (UID: "72aa6e74-5fcd-478f-a36f-871a0672a615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:42:22.072866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.072854 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:22.074405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.074384 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn" pod "72aa6e74-5fcd-478f-a36f-871a0672a615" (UID: "72aa6e74-5fcd-478f-a36f-871a0672a615"). InnerVolumeSpecName "kube-api-access-q5kzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:22.074540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.074527 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca" pod "72aa6e74-5fcd-478f-a36f-871a0672a615" (UID: "72aa6e74-5fcd-478f-a36f-871a0672a615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:22.174350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.174217 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kzn\" (UniqueName: \"kubernetes.io/projected/72aa6e74-5fcd-478f-a36f-871a0672a615-kube-api-access-q5kzn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:22.174350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.174251 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72aa6e74-5fcd-478f-a36f-871a0672a615-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:22.208598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.207971 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:22.208598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.208032 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:22.244903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.244875 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:22.245110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.245090 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:22.780590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.780549 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" Oct 30 22:42:22.780590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.780548 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7" event={"ID":"72aa6e74-5fcd-478f-a36f-871a0672a615","Type":"ContainerDied","Data":"242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a"} Oct 30 22:42:22.780864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:22.780628 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242cb5b5aa33abae7410f022b46715187d616459409a47b8ff3c0312f16f0d9a" Oct 30 22:42:23.208031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:23.207996 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:23.208327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:23.208298 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:23.245056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:23.245026 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:23.245269 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:23.245248 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:24.200599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:24.197902 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:24.200599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:24.197962 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:24.245175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:24.245140 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:24.245344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:24.245193 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:25.196102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:25.196057 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:25.196315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:25.196113 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:25.245241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:25.245207 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:25.245376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:25.245269 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:26.195515 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:26.195477 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:42:26.195699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:26.195543 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 22:42:26.244761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:26.244710 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:42:26.244976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:26.244780 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 22:42:26.651493 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:42:26.651429 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:42:28.236771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:28.236705 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:28.237404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:28.237347 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:28.284711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:28.284679 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:28.284861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:28.284728 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:29.278789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.278751 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:29.279219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.278819 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:29.325754 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.325707 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:29.325933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.325786 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:29.400403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.400372 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:29.401196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.401169 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:29.416555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.416524 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:29.416942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.416897 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:29.417668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.417647 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" Oct 30 22:42:29.487642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:29.487622 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" Oct 30 22:42:33.023753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.023720 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb"] Oct 30 22:42:33.024205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.024112 4292 cpu_manager.go:414] "containerMap: removing container" podUID="72aa6e74-5fcd-478f-a36f-871a0672a615" containerName="revoke-keys" Oct 30 22:42:33.024205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.024122 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72aa6e74-5fcd-478f-a36f-871a0672a615" containerName="revoke-keys" Oct 30 22:42:33.024205 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.024186 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="72aa6e74-5fcd-478f-a36f-871a0672a615" containerName="revoke-keys" Oct 30 22:42:33.027404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.027386 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121562 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7rd\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-kube-api-access-hx7rd\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121674 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121721 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121825 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121847 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121901 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121925 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.121969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.121949 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed66a5-595c-4751-be32-3e88be1d43d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223212 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223247 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223275 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223305 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223342 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed66a5-595c-4751-be32-3e88be1d43d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223377 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7rd\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-kube-api-access-hx7rd\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223427 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223450 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223662 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223707 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223758 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.223903 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223810 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.224031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.223917 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.224246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.224228 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed66a5-595c-4751-be32-3e88be1d43d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.225753 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.225729 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.225885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.225869 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.245661 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.245635 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb"] Oct 30 22:42:33.286827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.286775 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.287019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.286996 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7rd\" (UniqueName: \"kubernetes.io/projected/27ed66a5-595c-4751-be32-3e88be1d43d1-kube-api-access-hx7rd\") pod \"maas-default-gateway-openshift-default-76597d6855-md2fb\" (UID: \"27ed66a5-595c-4751-be32-3e88be1d43d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.339667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.339646 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:33.777286 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:42:33.777251 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ed66a5_595c_4751_be32_3e88be1d43d1.slice/crio-4920dacecdc50c57959256fa743c6babcbba48ef3edfaffbe21adf679d65a1fe WatchSource:0}: Error finding container 4920dacecdc50c57959256fa743c6babcbba48ef3edfaffbe21adf679d65a1fe: Status 404 returned error can't find the container with id 4920dacecdc50c57959256fa743c6babcbba48ef3edfaffbe21adf679d65a1fe Oct 30 22:42:33.777637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.777612 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb"] Oct 30 22:42:33.779412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.779378 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:33.779510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.779455 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:33.779510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.779496 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:33.855731 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.855683 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" event={"ID":"27ed66a5-595c-4751-be32-3e88be1d43d1","Type":"ContainerStarted","Data":"1d8f643afd9a8140d936c07008a684e8b69794a410ecfd070931da43d0c62601"} Oct 30 22:42:33.855876 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.855757 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" event={"ID":"27ed66a5-595c-4751-be32-3e88be1d43d1","Type":"ContainerStarted","Data":"4920dacecdc50c57959256fa743c6babcbba48ef3edfaffbe21adf679d65a1fe"} Oct 30 22:42:33.984029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:33.983764 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podStartSLOduration=1.983711727 podStartE2EDuration="1.983711727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:33.982237346 +0000 UTC m=+2881.951056911" watchObservedRunningTime="2026-10-30 22:42:33.983711727 +0000 UTC m=+2881.952531292" Oct 30 22:42:34.340811 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:34.340775 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:34.341918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:34.341894 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:34.342055 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:34.341940 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:35.340866 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:35.340831 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:35.341338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:35.340937 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:36.340865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:36.340829 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:36.341054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:36.340886 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:37.340280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.340245 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:37.340467 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.340298 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:37.942992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.942954 4292 scope.go:122] "RemoveContainer" containerID="888d23cd81d89d473216082b938e923e0bb081fbdaf6759c00ec334e949d24e9" Oct 30 22:42:37.951693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.951670 4292 scope.go:122] "RemoveContainer" containerID="f7ad75fbc8c8f36d80ee45a0329b014eb93d37c743e82ff6ecb7ad7017e60f8c" Oct 30 22:42:37.959601 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.959568 4292 scope.go:122] "RemoveContainer" containerID="cdad92dace2347b70db0cf77006313b38cf9bb2c2fa7bbc2de03778a83aeb444" Oct 30 22:42:37.966978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.966955 4292 scope.go:122] "RemoveContainer" containerID="fb00451b279fb2eccf28c355b7d2506c5cec11c7474ba5d57c8e233de1be9853" Oct 30 22:42:37.974704 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:37.974684 4292 scope.go:122] "RemoveContainer" containerID="c32df4bf6a73e998afb891a92ac17e9e422c16897521a563937b9a54e6de748b" Oct 30 22:42:38.340431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:38.340398 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:38.340632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:38.340460 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:39.340702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:39.340666 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:39.341098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:39.340714 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:40.340247 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:40.340210 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:42:40.340406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:40.340282 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 22:42:41.985968 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:41.985935 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:41.986424 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:41.986398 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:41.987222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:42:41.987206 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" Oct 30 22:42:56.638668 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:42:56.638626 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:43:25.056254 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:25.056216 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9"] Oct 30 22:43:25.106404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:25.106377 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d8ad-9v9x9"] Oct 30 22:43:26.144416 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:26.144377 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf" path="/var/lib/kubelet/pods/ffeb67a1-bd4f-4a78-9f1a-8ca46ce3e0bf/volumes" Oct 30 22:43:26.640059 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:43:26.640017 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:43:38.053996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:38.053965 4292 scope.go:122] "RemoveContainer" containerID="8c096a38f271c3d0cb3f9071c05f84fcff9dc0b371f6f2dbcf4f94e67bad2073" Oct 30 22:43:38.058114 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:38.058092 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv"] Oct 30 22:43:38.063136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:38.063115 4292 scope.go:122] "RemoveContainer" containerID="cbcd267d6f6ad3206ded1e5efda702c4ce4707243107d22bde32c1a0a13f2475" Oct 30 22:43:38.108353 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:38.108328 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f073dff1-kxhlv"] Oct 30 22:43:38.145707 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:38.145681 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cce9ff3-8bb7-49c4-a47a-e4fb070d3594" path="/var/lib/kubelet/pods/0cce9ff3-8bb7-49c4-a47a-e4fb070d3594/volumes" Oct 30 22:43:56.639993 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:43:56.639955 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:43:58.167388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.167353 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp"] Oct 30 22:43:58.172848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.172829 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.230952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.230924 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp"] Oct 30 22:43:58.278631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.278604 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:43:58.278757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.278632 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:43:58.278757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.278638 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:43:58.278757 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.278669 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:43:58.329511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.329480 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.329655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.329560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.329702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.329682 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbps\" (UniqueName: \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.430423 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.430363 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.430533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.430426 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbps\" (UniqueName: \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.430533 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.430478 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.431077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.431056 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.432799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.432780 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.494411 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.494382 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbps\" (UniqueName: \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\") pod \"maas-api-revoke-sub-2e1f5b5f-zlqbp\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:58.785728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:58.785657 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:43:59.160469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:59.160439 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp"] Oct 30 22:43:59.162065 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:43:59.162034 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79d38aa_ff5f_427c_b8c4_d22d71130475.slice/crio-5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779 WatchSource:0}: Error finding container 5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779: Status 404 returned error can't find the container with id 5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779 Oct 30 22:43:59.306626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:43:59.306595 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" event={"ID":"e79d38aa-ff5f-427c-b8c4-d22d71130475","Type":"ContainerStarted","Data":"5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779"} Oct 30 22:44:00.319982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:00.319938 4292 generic.go:356] "Generic (PLEG): container finished" podID="e79d38aa-ff5f-427c-b8c4-d22d71130475" containerID="e2c123aa4139344151ff9b9507bdb0f46b2994602691cfd9bc9557b1b5343cef" exitCode=0 Oct 30 22:44:00.320358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:00.320023 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" event={"ID":"e79d38aa-ff5f-427c-b8c4-d22d71130475","Type":"ContainerDied","Data":"e2c123aa4139344151ff9b9507bdb0f46b2994602691cfd9bc9557b1b5343cef"} Oct 30 22:44:01.461537 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.461513 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:44:01.558035 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.558007 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\") pod \"e79d38aa-ff5f-427c-b8c4-d22d71130475\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " Oct 30 22:44:01.558175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.558048 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\" (UniqueName: \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\") pod \"e79d38aa-ff5f-427c-b8c4-d22d71130475\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " Oct 30 22:44:01.558175 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.558166 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\") pod \"e79d38aa-ff5f-427c-b8c4-d22d71130475\" (UID: \"e79d38aa-ff5f-427c-b8c4-d22d71130475\") " Oct 30 22:44:01.558329 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.558306 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca" pod "e79d38aa-ff5f-427c-b8c4-d22d71130475" (UID: "e79d38aa-ff5f-427c-b8c4-d22d71130475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:01.558510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.558486 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:01.560031 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.560009 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca" pod "e79d38aa-ff5f-427c-b8c4-d22d71130475" (UID: "e79d38aa-ff5f-427c-b8c4-d22d71130475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:01.560137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.560055 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps" pod "e79d38aa-ff5f-427c-b8c4-d22d71130475" (UID: "e79d38aa-ff5f-427c-b8c4-d22d71130475"). InnerVolumeSpecName "kube-api-access-ldbps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:01.659025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.658960 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d38aa-ff5f-427c-b8c4-d22d71130475-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:01.659025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:01.658986 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbps\" (UniqueName: \"kubernetes.io/projected/e79d38aa-ff5f-427c-b8c4-d22d71130475-kube-api-access-ldbps\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:02.331850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:02.331821 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" Oct 30 22:44:02.331850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:02.331812 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp" event={"ID":"e79d38aa-ff5f-427c-b8c4-d22d71130475","Type":"ContainerDied","Data":"5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779"} Oct 30 22:44:02.332108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:02.331889 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c71190255b66024c00cbc62ba0a14957bbcd82d31748a4d7a755ce9cfc3f779" Oct 30 22:44:10.597405 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.597368 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp"] Oct 30 22:44:10.597844 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.597830 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e79d38aa-ff5f-427c-b8c4-d22d71130475" containerName="revoke-keys" Oct 30 22:44:10.597899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.597845 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d38aa-ff5f-427c-b8c4-d22d71130475" containerName="revoke-keys" Oct 30 22:44:10.597936 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.597918 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79d38aa-ff5f-427c-b8c4-d22d71130475" containerName="revoke-keys" Oct 30 22:44:10.601098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.601078 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.639421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.639395 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.639588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.639461 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4cn\" (UniqueName: \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.639659 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.639586 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.666678 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.666648 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp"] Oct 30 22:44:10.669965 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.669932 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:44:10.714504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.714480 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:44:10.714823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.714803 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:44:10.714894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.714846 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:44:10.740830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.740809 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.740930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.740870 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4cn\" (UniqueName: \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.740930 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.740899 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.741564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.741543 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.743236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.743217 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.805453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.805429 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4cn\" (UniqueName: \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\") pod \"maas-api-revoke-sub-ec843018-ftlcp\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:10.914152 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:10.914081 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:11.294238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:11.294205 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp"] Oct 30 22:44:11.295742 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:44:11.295713 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408f0fe1_b7ca_451f_a5f3_4cb59e989b40.slice/crio-cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c WatchSource:0}: Error finding container cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c: Status 404 returned error can't find the container with id cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c Oct 30 22:44:11.376504 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:11.376474 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" event={"ID":"408f0fe1-b7ca-451f-a5f3-4cb59e989b40","Type":"ContainerStarted","Data":"cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c"} Oct 30 22:44:12.382917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:12.382880 4292 generic.go:356] "Generic (PLEG): container finished" podID="408f0fe1-b7ca-451f-a5f3-4cb59e989b40" containerID="6b5d1c9117d7e197cead7ff69da52115adacab1fc61a027a2e76424c7649cbd2" exitCode=0 Oct 30 22:44:12.383307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:12.382933 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" event={"ID":"408f0fe1-b7ca-451f-a5f3-4cb59e989b40","Type":"ContainerDied","Data":"6b5d1c9117d7e197cead7ff69da52115adacab1fc61a027a2e76424c7649cbd2"} Oct 30 22:44:13.516888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.516863 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:13.568136 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.568107 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\") pod \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " Oct 30 22:44:13.568272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.568149 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\" (UniqueName: \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\") pod \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " Oct 30 22:44:13.568272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.568210 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\") pod \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\" (UID: \"408f0fe1-b7ca-451f-a5f3-4cb59e989b40\") " Oct 30 22:44:13.568501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.568474 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca" pod "408f0fe1-b7ca-451f-a5f3-4cb59e989b40" (UID: "408f0fe1-b7ca-451f-a5f3-4cb59e989b40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:13.570173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.570142 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca" pod "408f0fe1-b7ca-451f-a5f3-4cb59e989b40" (UID: "408f0fe1-b7ca-451f-a5f3-4cb59e989b40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:13.570173 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.570164 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn" pod "408f0fe1-b7ca-451f-a5f3-4cb59e989b40" (UID: "408f0fe1-b7ca-451f-a5f3-4cb59e989b40"). InnerVolumeSpecName "kube-api-access-4z4cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:13.669219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.669139 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:13.669219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.669167 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4cn\" (UniqueName: \"kubernetes.io/projected/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-kube-api-access-4z4cn\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:13.669219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:13.669181 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408f0fe1-b7ca-451f-a5f3-4cb59e989b40-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:14.393668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:14.393641 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" Oct 30 22:44:14.393668 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:14.393661 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp" event={"ID":"408f0fe1-b7ca-451f-a5f3-4cb59e989b40","Type":"ContainerDied","Data":"cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c"} Oct 30 22:44:14.393884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:14.393733 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc320a075d72c2b0fed8e452be7de60e11b37b069f87034e7e73e2bf04c5f70c" Oct 30 22:44:26.639214 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:44:26.639166 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:44:30.451611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:30.449665 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:44:30.451611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:30.449747 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:44:30.460826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:30.460673 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:44:30.460826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:30.460737 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:44:38.084935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:38.084897 4292 scope.go:122] "RemoveContainer" containerID="cfe1fa5c45562a6eaafac27fd8a609d32f717dd75a4e10b71c5b8aef3cadc6ab" Oct 30 22:44:38.095692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:38.095548 4292 scope.go:122] "RemoveContainer" containerID="648dd1d23af7d1c4543bef24d6869617da80be3aac115d6615403d62c644c666" Oct 30 22:44:46.178094 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.178021 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl"] Oct 30 22:44:46.178460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.178419 4292 cpu_manager.go:414] "containerMap: removing container" podUID="408f0fe1-b7ca-451f-a5f3-4cb59e989b40" containerName="revoke-keys" Oct 30 22:44:46.178460 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.178431 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408f0fe1-b7ca-451f-a5f3-4cb59e989b40" containerName="revoke-keys" Oct 30 22:44:46.178539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.178495 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="408f0fe1-b7ca-451f-a5f3-4cb59e989b40" containerName="revoke-keys" Oct 30 22:44:46.182926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.182908 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.241882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.241847 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl"] Oct 30 22:44:46.271050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.271010 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.271223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.271082 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rt6s\" (UniqueName: \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.271306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.271278 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.289755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.289732 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:44:46.289883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.289819 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:44:46.289883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.289850 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:44:46.289883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.289859 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:44:46.372291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.372265 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.372436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.372309 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.372436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.372335 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rt6s\" (UniqueName: \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.372972 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.372952 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.374581 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.374559 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.435859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.435808 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rt6s\" (UniqueName: \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\") pod \"maas-api-revoke-sub-046e1343-69bbl\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.495642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.495609 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:46.870584 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:46.870552 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl"] Oct 30 22:44:46.872122 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:44:46.872095 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5bf0a2_7fe4_4eb2_8959_186e79f40553.slice/crio-d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece WatchSource:0}: Error finding container d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece: Status 404 returned error can't find the container with id d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece Oct 30 22:44:47.565084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:47.565048 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" event={"ID":"3f5bf0a2-7fe4-4eb2-8959-186e79f40553","Type":"ContainerStarted","Data":"85bfc41bc49ff5e418bf459c76f097a14481fa39dd487a7bb0e5c61335500e07"} Oct 30 22:44:47.565478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:47.565108 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" event={"ID":"3f5bf0a2-7fe4-4eb2-8959-186e79f40553","Type":"ContainerStarted","Data":"d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece"} Oct 30 22:44:47.686120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:47.686070 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" podStartSLOduration=1.6860559450000001 podStartE2EDuration="1.686055945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:47.685838656 +0000 UTC m=+3015.654658242" watchObservedRunningTime="2026-10-30 22:44:47.686055945 +0000 UTC m=+3015.654875506" Oct 30 22:44:48.571798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:48.571764 4292 generic.go:356] "Generic (PLEG): container finished" podID="3f5bf0a2-7fe4-4eb2-8959-186e79f40553" containerID="85bfc41bc49ff5e418bf459c76f097a14481fa39dd487a7bb0e5c61335500e07" exitCode=0 Oct 30 22:44:48.572191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:48.571828 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" event={"ID":"3f5bf0a2-7fe4-4eb2-8959-186e79f40553","Type":"ContainerDied","Data":"85bfc41bc49ff5e418bf459c76f097a14481fa39dd487a7bb0e5c61335500e07"} Oct 30 22:44:49.721122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.721097 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:49.803642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.803614 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\") pod \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " Oct 30 22:44:49.803766 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.803734 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\") pod \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " Oct 30 22:44:49.803827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.803787 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\" (UniqueName: \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\") pod \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\" (UID: \"3f5bf0a2-7fe4-4eb2-8959-186e79f40553\") " Oct 30 22:44:49.803949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.803928 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca" pod "3f5bf0a2-7fe4-4eb2-8959-186e79f40553" (UID: "3f5bf0a2-7fe4-4eb2-8959-186e79f40553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:49.804036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.804017 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:49.805723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.805703 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca" pod "3f5bf0a2-7fe4-4eb2-8959-186e79f40553" (UID: "3f5bf0a2-7fe4-4eb2-8959-186e79f40553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:49.805942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.805921 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s" pod "3f5bf0a2-7fe4-4eb2-8959-186e79f40553" (UID: "3f5bf0a2-7fe4-4eb2-8959-186e79f40553"). InnerVolumeSpecName "kube-api-access-7rt6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:49.905531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.905458 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rt6s\" (UniqueName: \"kubernetes.io/projected/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-kube-api-access-7rt6s\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:49.905531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:49.905484 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5bf0a2-7fe4-4eb2-8959-186e79f40553-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:50.583272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:50.583244 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" Oct 30 22:44:50.583272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:50.583253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl" event={"ID":"3f5bf0a2-7fe4-4eb2-8959-186e79f40553","Type":"ContainerDied","Data":"d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece"} Oct 30 22:44:50.583559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:50.583296 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71e19f5237ba999522670f1d68c501807a416ad12b6b696d278aea82cb20ece" Oct 30 22:44:56.639029 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:44:56.638976 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:44:56.996144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:56.996070 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk"] Oct 30 22:44:56.996532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:56.996516 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3f5bf0a2-7fe4-4eb2-8959-186e79f40553" containerName="revoke-keys" Oct 30 22:44:56.996532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:56.996533 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5bf0a2-7fe4-4eb2-8959-186e79f40553" containerName="revoke-keys" Oct 30 22:44:56.996677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:56.996633 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5bf0a2-7fe4-4eb2-8959-186e79f40553" containerName="revoke-keys" Oct 30 22:44:56.999685 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:56.999670 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.055232 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.055201 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:44:57.055401 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.055385 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:44:57.076370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.076346 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkxj\" (UniqueName: \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.076512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.076407 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.076582 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.076501 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.106090 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.106061 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:44:57.106196 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.106095 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:44:57.108281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.108260 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk"] Oct 30 22:44:57.177098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.177062 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.177234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.177167 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkxj\" (UniqueName: \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.177291 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.177242 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.177761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.177740 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.179471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.179452 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.240682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.240660 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkxj\" (UniqueName: \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\") pod \"maas-api-revoke-sub-d3fba056-4m9lk\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.312284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.312251 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:57.688476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:57.688446 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk"] Oct 30 22:44:57.690025 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:44:57.689994 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf762fb2_587e_4d2c_a8b0_81ef21c3201a.slice/crio-f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08 WatchSource:0}: Error finding container f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08: Status 404 returned error can't find the container with id f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08 Oct 30 22:44:58.623385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:58.623353 4292 generic.go:356] "Generic (PLEG): container finished" podID="bf762fb2-587e-4d2c-a8b0-81ef21c3201a" containerID="a8c2a9b229d2d4b2bc1c37935268a78ba8e451a949d5c4bcb6733d49bcf4c943" exitCode=0 Oct 30 22:44:58.623526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:58.623409 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" event={"ID":"bf762fb2-587e-4d2c-a8b0-81ef21c3201a","Type":"ContainerDied","Data":"a8c2a9b229d2d4b2bc1c37935268a78ba8e451a949d5c4bcb6733d49bcf4c943"} Oct 30 22:44:58.623526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:58.623451 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" event={"ID":"bf762fb2-587e-4d2c-a8b0-81ef21c3201a","Type":"ContainerStarted","Data":"f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08"} Oct 30 22:44:59.790056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.790021 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:44:59.899609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.899517 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\") pod \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " Oct 30 22:44:59.899780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.899659 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\" (UniqueName: \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\") pod \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " Oct 30 22:44:59.899780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.899733 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\") pod \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\" (UID: \"bf762fb2-587e-4d2c-a8b0-81ef21c3201a\") " Oct 30 22:44:59.900167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.900118 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca" pod "bf762fb2-587e-4d2c-a8b0-81ef21c3201a" (UID: "bf762fb2-587e-4d2c-a8b0-81ef21c3201a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:59.901951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.901921 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca" pod "bf762fb2-587e-4d2c-a8b0-81ef21c3201a" (UID: "bf762fb2-587e-4d2c-a8b0-81ef21c3201a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:59.902107 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:44:59.902087 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj" pod "bf762fb2-587e-4d2c-a8b0-81ef21c3201a" (UID: "bf762fb2-587e-4d2c-a8b0-81ef21c3201a"). InnerVolumeSpecName "kube-api-access-9rkxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:00.000806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.000767 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rkxj\" (UniqueName: \"kubernetes.io/projected/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-kube-api-access-9rkxj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:00.000806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.000794 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:00.000806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.000804 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf762fb2-587e-4d2c-a8b0-81ef21c3201a-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:00.176387 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.176298 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:00.176945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.176919 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bf762fb2-587e-4d2c-a8b0-81ef21c3201a" containerName="revoke-keys" Oct 30 22:45:00.176945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.176946 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf762fb2-587e-4d2c-a8b0-81ef21c3201a" containerName="revoke-keys" Oct 30 22:45:00.177132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.177108 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf762fb2-587e-4d2c-a8b0-81ef21c3201a" containerName="revoke-keys" Oct 30 22:45:00.180785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.180761 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.240143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.240086 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:00.304987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.304950 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9dl\" (UniqueName: \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.305249 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.305225 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.406133 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.406088 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9dl\" (UniqueName: \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.406486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.406456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.407496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.407471 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.470221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.470159 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9dl\" (UniqueName: \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\") pod \"maas-api-key-cleanup-29890005-5mb6h\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.496740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.496711 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:00.659521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.658358 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" event={"ID":"bf762fb2-587e-4d2c-a8b0-81ef21c3201a","Type":"ContainerDied","Data":"f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08"} Oct 30 22:45:00.659521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.658425 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f32640b4bd36a6c3fb72e71372e58a689129206313814c2da13e58e847010b08" Oct 30 22:45:00.659521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:00.658507 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk" Oct 30 22:45:01.044595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:01.018267 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:01.045799 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:45:01.045111 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18983ae2_f936_4e25_b3d3_046d25852e4a.slice/crio-2d8799a7b35c6a1a613e3e436ee9641d95c7daa1d3ce1bf0d3d3615565f46849 WatchSource:0}: Error finding container 2d8799a7b35c6a1a613e3e436ee9641d95c7daa1d3ce1bf0d3d3615565f46849: Status 404 returned error can't find the container with id 2d8799a7b35c6a1a613e3e436ee9641d95c7daa1d3ce1bf0d3d3615565f46849 Oct 30 22:45:01.666722 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:01.666684 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerStarted","Data":"299c5e7b847228aa2e498411b872afa85d1d765287e8644713a2d76de214ab3d"} Oct 30 22:45:01.666924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:01.666737 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerStarted","Data":"2d8799a7b35c6a1a613e3e436ee9641d95c7daa1d3ce1bf0d3d3615565f46849"} Oct 30 22:45:01.798653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:01.791526 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" podStartSLOduration=1.791500477 podStartE2EDuration="1.791500477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.788939857 +0000 UTC m=+3029.757759422" watchObservedRunningTime="2026-10-30 22:45:01.791500477 +0000 UTC m=+3029.760320042" Oct 30 22:45:02.674183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:02.674146 4292 generic.go:356] "Generic (PLEG): container finished" podID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerID="299c5e7b847228aa2e498411b872afa85d1d765287e8644713a2d76de214ab3d" exitCode=60 Oct 30 22:45:02.674643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:02.674222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerDied","Data":"299c5e7b847228aa2e498411b872afa85d1d765287e8644713a2d76de214ab3d"} Oct 30 22:45:02.674643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:02.674463 4292 scope.go:122] "RemoveContainer" containerID="299c5e7b847228aa2e498411b872afa85d1d765287e8644713a2d76de214ab3d" Oct 30 22:45:03.681174 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:03.681142 4292 generic.go:356] "Generic (PLEG): container finished" podID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerID="a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f" exitCode=60 Oct 30 22:45:03.681654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:03.681181 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerDied","Data":"a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f"} Oct 30 22:45:03.681654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:03.681226 4292 scope.go:122] "RemoveContainer" containerID="299c5e7b847228aa2e498411b872afa85d1d765287e8644713a2d76de214ab3d" Oct 30 22:45:03.681654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:03.681518 4292 scope.go:122] "RemoveContainer" containerID="a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f" Oct 30 22:45:03.681830 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:45:03.681753 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-5mb6h_odh-ai-gateway-infra(18983ae2-f936-4e25-b3d3-046d25852e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" Oct 30 22:45:04.687527 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:04.687497 4292 scope.go:122] "RemoveContainer" containerID="a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f" Oct 30 22:45:04.687948 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:45:04.687736 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-5mb6h_odh-ai-gateway-infra(18983ae2-f936-4e25-b3d3-046d25852e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" Oct 30 22:45:15.139814 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:15.139781 4292 scope.go:122] "RemoveContainer" containerID="a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f" Oct 30 22:45:15.741764 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:15.741726 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerStarted","Data":"67617e5b3672dd1ba4e274029c5299ab5b728a4d31a5cf23a6683651a9a4cccd"} Oct 30 22:45:16.751758 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:16.751720 4292 generic.go:356] "Generic (PLEG): container finished" podID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerID="67617e5b3672dd1ba4e274029c5299ab5b728a4d31a5cf23a6683651a9a4cccd" exitCode=60 Oct 30 22:45:16.752260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:16.751805 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerDied","Data":"67617e5b3672dd1ba4e274029c5299ab5b728a4d31a5cf23a6683651a9a4cccd"} Oct 30 22:45:16.752260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:16.751870 4292 scope.go:122] "RemoveContainer" containerID="a7764a6344e12a1cf25362f3c373b3ab29537f467fe96832c3c91f0620f20f6f" Oct 30 22:45:16.752418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:16.752257 4292 scope.go:122] "RemoveContainer" containerID="67617e5b3672dd1ba4e274029c5299ab5b728a4d31a5cf23a6683651a9a4cccd" Oct 30 22:45:16.752499 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:45:16.752477 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-5mb6h_odh-ai-gateway-infra(18983ae2-f936-4e25-b3d3-046d25852e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" Oct 30 22:45:16.879714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:16.879673 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:18.163801 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.163766 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:18.279203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.279164 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\" (UniqueName: \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\") pod \"18983ae2-f936-4e25-b3d3-046d25852e4a\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " Oct 30 22:45:18.279366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.279272 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\") pod \"18983ae2-f936-4e25-b3d3-046d25852e4a\" (UID: \"18983ae2-f936-4e25-b3d3-046d25852e4a\") " Oct 30 22:45:18.280256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.280236 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca" pod "18983ae2-f936-4e25-b3d3-046d25852e4a" (UID: "18983ae2-f936-4e25-b3d3-046d25852e4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.281448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.281428 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl" pod "18983ae2-f936-4e25-b3d3-046d25852e4a" (UID: "18983ae2-f936-4e25-b3d3-046d25852e4a"). InnerVolumeSpecName "kube-api-access-bv9dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.382255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.382136 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9dl\" (UniqueName: \"kubernetes.io/projected/18983ae2-f936-4e25-b3d3-046d25852e4a-kube-api-access-bv9dl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.382255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.382166 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18983ae2-f936-4e25-b3d3-046d25852e4a-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.776637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.776540 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" event={"ID":"18983ae2-f936-4e25-b3d3-046d25852e4a","Type":"ContainerDied","Data":"2d8799a7b35c6a1a613e3e436ee9641d95c7daa1d3ce1bf0d3d3615565f46849"} Oct 30 22:45:18.776637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.776626 4292 scope.go:122] "RemoveContainer" containerID="67617e5b3672dd1ba4e274029c5299ab5b728a4d31a5cf23a6683651a9a4cccd" Oct 30 22:45:18.776849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.776557 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h" Oct 30 22:45:18.956523 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.956410 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:18.962546 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:18.962506 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-5mb6h"] Oct 30 22:45:20.145684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:20.145653 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" path="/var/lib/kubelet/pods/18983ae2-f936-4e25-b3d3-046d25852e4a/volumes" Oct 30 22:45:26.638497 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:45:26.638447 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:45:35.222871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.222834 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd"] Oct 30 22:45:35.223288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223231 4292 cpu_manager.go:414] "containerMap: removing container" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.223288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223243 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.223288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223257 4292 cpu_manager.go:414] "containerMap: removing container" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.223288 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223263 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.223430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223343 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.223430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.223359 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:45:35.226431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.226415 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.242123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.242095 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.242263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.242130 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.242263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.242148 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmb6\" (UniqueName: \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.281840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.281816 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:45:35.331891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.331857 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:45:35.331891 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.331879 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:45:35.333990 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.333970 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd"] Oct 30 22:45:35.343468 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.343442 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.343569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.343473 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.343569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.343494 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmb6\" (UniqueName: \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.345938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.345870 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.382784 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.382763 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:45:35.384238 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.384220 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.407360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.407336 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmb6\" (UniqueName: \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\") pod \"maas-api-revoke-sub-ae594979-xlrhd\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.538551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.538487 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:35.912885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:35.912867 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd"] Oct 30 22:45:35.915236 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:45:35.915207 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469f9b88_2098_485e_b0de_e31d3f8f84cf.slice/crio-c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad WatchSource:0}: Error finding container c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad: Status 404 returned error can't find the container with id c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad Oct 30 22:45:36.866857 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:36.866822 4292 generic.go:356] "Generic (PLEG): container finished" podID="469f9b88-2098-485e-b0de-e31d3f8f84cf" containerID="2dacc1d92afc8cdac5f4887eab8d3b29d0f71799434abab05928b1464df27c8c" exitCode=0 Oct 30 22:45:36.867253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:36.866912 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" event={"ID":"469f9b88-2098-485e-b0de-e31d3f8f84cf","Type":"ContainerDied","Data":"2dacc1d92afc8cdac5f4887eab8d3b29d0f71799434abab05928b1464df27c8c"} Oct 30 22:45:36.867253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:36.867028 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" event={"ID":"469f9b88-2098-485e-b0de-e31d3f8f84cf","Type":"ContainerStarted","Data":"c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad"} Oct 30 22:45:38.007103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.007076 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:38.070526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.070497 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\") pod \"469f9b88-2098-485e-b0de-e31d3f8f84cf\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " Oct 30 22:45:38.070715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.070546 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\" (UniqueName: \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\") pod \"469f9b88-2098-485e-b0de-e31d3f8f84cf\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " Oct 30 22:45:38.070715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.070626 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\") pod \"469f9b88-2098-485e-b0de-e31d3f8f84cf\" (UID: \"469f9b88-2098-485e-b0de-e31d3f8f84cf\") " Oct 30 22:45:38.071007 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.070982 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca" pod "469f9b88-2098-485e-b0de-e31d3f8f84cf" (UID: "469f9b88-2098-485e-b0de-e31d3f8f84cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:38.072494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.072470 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca" pod "469f9b88-2098-485e-b0de-e31d3f8f84cf" (UID: "469f9b88-2098-485e-b0de-e31d3f8f84cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:38.072569 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.072556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6" pod "469f9b88-2098-485e-b0de-e31d3f8f84cf" (UID: "469f9b88-2098-485e-b0de-e31d3f8f84cf"). InnerVolumeSpecName "kube-api-access-gtmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:38.117486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.117461 4292 scope.go:122] "RemoveContainer" containerID="6740fe8a563370f06b00bc8caa1e5ed2abd86710c326ab62ca7f59bc36c0b886" Oct 30 22:45:38.125894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.125871 4292 scope.go:122] "RemoveContainer" containerID="8624bbbe942780ac0987638dcf8313ac5088f03b3b4389dadb33a8e7c4344978" Oct 30 22:45:38.171413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.171385 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:38.171413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.171408 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmb6\" (UniqueName: \"kubernetes.io/projected/469f9b88-2098-485e-b0de-e31d3f8f84cf-kube-api-access-gtmb6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:38.171413 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.171418 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469f9b88-2098-485e-b0de-e31d3f8f84cf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:38.879364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.879337 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" Oct 30 22:45:38.879364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.879361 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd" event={"ID":"469f9b88-2098-485e-b0de-e31d3f8f84cf","Type":"ContainerDied","Data":"c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad"} Oct 30 22:45:38.879641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:45:38.879402 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c53d5219906c9dbec43dc8893798ccb195530390d83e99022052fcaab4e210ad" Oct 30 22:45:56.639452 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:45:56.639415 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:46:06.518355 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.518315 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb"] Oct 30 22:46:06.518995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.518972 4292 cpu_manager.go:414] "containerMap: removing container" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:46:06.519098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.518997 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:46:06.519098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.519016 4292 cpu_manager.go:414] "containerMap: removing container" podUID="469f9b88-2098-485e-b0de-e31d3f8f84cf" containerName="revoke-keys" Oct 30 22:46:06.519098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.519030 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469f9b88-2098-485e-b0de-e31d3f8f84cf" containerName="revoke-keys" Oct 30 22:46:06.519282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.519164 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="18983ae2-f936-4e25-b3d3-046d25852e4a" containerName="cleanup" Oct 30 22:46:06.519282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.519182 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="469f9b88-2098-485e-b0de-e31d3f8f84cf" containerName="revoke-keys" Oct 30 22:46:06.522446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.522424 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.547350 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.547317 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.547499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.547356 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.547499 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.547375 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6w5\" (UniqueName: \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.582492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.582462 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb"] Oct 30 22:46:06.629715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.629681 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:46:06.629715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.629689 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:46:06.629715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.629701 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:46:06.629932 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.629742 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:46:06.648473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.648449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.648632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.648488 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.648632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.648507 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6w5\" (UniqueName: \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.649219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.649201 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.650755 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.650738 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.711913 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.711890 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6w5\" (UniqueName: \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\") pod \"maas-api-revoke-sub-a4aef7d7-kkkfb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:06.834763 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:06.834731 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:07.211370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:07.211280 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb"] Oct 30 22:46:07.214973 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:46:07.214939 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8486d76_1373_45b6_9671_5474ce6569bb.slice/crio-f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d WatchSource:0}: Error finding container f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d: Status 404 returned error can't find the container with id f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d Oct 30 22:46:08.026043 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:08.025998 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" event={"ID":"e8486d76-1373-45b6-9671-5474ce6569bb","Type":"ContainerStarted","Data":"3347f60f8cb8a558d62675118027021b648300325bd1a6e6ca86b1e63936e153"} Oct 30 22:46:08.026518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:08.026058 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" event={"ID":"e8486d76-1373-45b6-9671-5474ce6569bb","Type":"ContainerStarted","Data":"f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d"} Oct 30 22:46:08.145963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:08.145917 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" podStartSLOduration=2.145903369 podStartE2EDuration="2.145903369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:08.145334811 +0000 UTC m=+3096.114154375" watchObservedRunningTime="2026-10-30 22:46:08.145903369 +0000 UTC m=+3096.114722933" Oct 30 22:46:09.033081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:09.033041 4292 generic.go:356] "Generic (PLEG): container finished" podID="e8486d76-1373-45b6-9671-5474ce6569bb" containerID="3347f60f8cb8a558d62675118027021b648300325bd1a6e6ca86b1e63936e153" exitCode=0 Oct 30 22:46:09.033543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:09.033130 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" event={"ID":"e8486d76-1373-45b6-9671-5474ce6569bb","Type":"ContainerDied","Data":"3347f60f8cb8a558d62675118027021b648300325bd1a6e6ca86b1e63936e153"} Oct 30 22:46:10.168812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.168788 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:10.174940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.174919 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\" (UniqueName: \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\") pod \"e8486d76-1373-45b6-9671-5474ce6569bb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " Oct 30 22:46:10.175004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.174987 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\") pod \"e8486d76-1373-45b6-9671-5474ce6569bb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " Oct 30 22:46:10.175057 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.175008 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\") pod \"e8486d76-1373-45b6-9671-5474ce6569bb\" (UID: \"e8486d76-1373-45b6-9671-5474ce6569bb\") " Oct 30 22:46:10.175318 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.175297 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca" pod "e8486d76-1373-45b6-9671-5474ce6569bb" (UID: "e8486d76-1373-45b6-9671-5474ce6569bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:10.176971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.176948 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca" pod "e8486d76-1373-45b6-9671-5474ce6569bb" (UID: "e8486d76-1373-45b6-9671-5474ce6569bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:10.176971 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.176953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5" pod "e8486d76-1373-45b6-9671-5474ce6569bb" (UID: "e8486d76-1373-45b6-9671-5474ce6569bb"). InnerVolumeSpecName "kube-api-access-pm6w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:10.276204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.276165 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6w5\" (UniqueName: \"kubernetes.io/projected/e8486d76-1373-45b6-9671-5474ce6569bb-kube-api-access-pm6w5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:10.276204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.276199 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:10.276204 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:10.276210 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8486d76-1373-45b6-9671-5474ce6569bb-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:11.045221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:11.045190 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" Oct 30 22:46:11.045402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:11.045225 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb" event={"ID":"e8486d76-1373-45b6-9671-5474ce6569bb","Type":"ContainerDied","Data":"f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d"} Oct 30 22:46:11.045402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:11.045279 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39666a67bc096f679df24569c48c6cc613adc8506dfa6a2a213b4b44c2aa07d" Oct 30 22:46:26.061335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:26.061302 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66"] Oct 30 22:46:26.111769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:26.111728 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3c24d18-frc66"] Oct 30 22:46:26.145320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:26.145287 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9599455-7d55-45fb-85f2-e0a303539ff1" path="/var/lib/kubelet/pods/f9599455-7d55-45fb-85f2-e0a303539ff1/volumes" Oct 30 22:46:26.642025 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:46:26.641980 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:46:37.352284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.352248 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r"] Oct 30 22:46:37.352785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.352683 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e8486d76-1373-45b6-9671-5474ce6569bb" containerName="revoke-keys" Oct 30 22:46:37.352785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.352695 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8486d76-1373-45b6-9671-5474ce6569bb" containerName="revoke-keys" Oct 30 22:46:37.352785 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.352777 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8486d76-1373-45b6-9671-5474ce6569bb" containerName="revoke-keys" Oct 30 22:46:37.355834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.355812 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.420016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.419992 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:46:37.448442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.448420 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.448561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.448464 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.448561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.448498 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88v9z\" (UniqueName: \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.461713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.461691 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:46:37.461815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.461750 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:46:37.461884 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.461871 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:46:37.463969 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.463949 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r"] Oct 30 22:46:37.549229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.549206 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.549328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.549247 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.549328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.549279 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88v9z\" (UniqueName: \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.549982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.549964 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.551679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.551657 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.612403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.612340 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88v9z\" (UniqueName: \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\") pod \"maas-api-revoke-sub-79805ca9-wvj7r\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:37.668150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:37.668127 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:38.041108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:38.041082 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r"] Oct 30 22:46:38.042549 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:46:38.042512 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd552b3_a653_451b_b473_3c8a972ef789.slice/crio-34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3 WatchSource:0}: Error finding container 34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3: Status 404 returned error can't find the container with id 34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3 Oct 30 22:46:38.158374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:38.158348 4292 scope.go:122] "RemoveContainer" containerID="301f6d02efeb1dcaaf4be0dd0bfa0f7ea877efcd8095236c894cc22b58aa7d67" Oct 30 22:46:38.166510 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:38.166495 4292 scope.go:122] "RemoveContainer" containerID="7ee56c17d074d9f2bb62ca3416446ffd67389c12b283bdeb0407d9b3c90c853b" Oct 30 22:46:38.204478 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:38.204453 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" event={"ID":"0bd552b3-a653-451b-b473-3c8a972ef789","Type":"ContainerStarted","Data":"34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3"} Oct 30 22:46:39.211002 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:39.210913 4292 generic.go:356] "Generic (PLEG): container finished" podID="0bd552b3-a653-451b-b473-3c8a972ef789" containerID="c9304617ebcc52eb67a5bc2e2bf7dd8a3e3f885bcbf3923ce48b6fe05121a2f0" exitCode=0 Oct 30 22:46:39.211381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:39.210999 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" event={"ID":"0bd552b3-a653-451b-b473-3c8a972ef789","Type":"ContainerDied","Data":"c9304617ebcc52eb67a5bc2e2bf7dd8a3e3f885bcbf3923ce48b6fe05121a2f0"} Oct 30 22:46:40.347220 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.347188 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:40.476016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.475927 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\") pod \"0bd552b3-a653-451b-b473-3c8a972ef789\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " Oct 30 22:46:40.476016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.475983 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\" (UniqueName: \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\") pod \"0bd552b3-a653-451b-b473-3c8a972ef789\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " Oct 30 22:46:40.476016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.476014 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\") pod \"0bd552b3-a653-451b-b473-3c8a972ef789\" (UID: \"0bd552b3-a653-451b-b473-3c8a972ef789\") " Oct 30 22:46:40.476324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.476300 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca" pod "0bd552b3-a653-451b-b473-3c8a972ef789" (UID: "0bd552b3-a653-451b-b473-3c8a972ef789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:40.478120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.478090 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca" pod "0bd552b3-a653-451b-b473-3c8a972ef789" (UID: "0bd552b3-a653-451b-b473-3c8a972ef789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:40.478257 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.478147 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z" pod "0bd552b3-a653-451b-b473-3c8a972ef789" (UID: "0bd552b3-a653-451b-b473-3c8a972ef789"). InnerVolumeSpecName "kube-api-access-88v9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:40.576770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.576736 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:40.576770 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.576766 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88v9z\" (UniqueName: \"kubernetes.io/projected/0bd552b3-a653-451b-b473-3c8a972ef789-kube-api-access-88v9z\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:40.576941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:40.576777 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd552b3-a653-451b-b473-3c8a972ef789-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:41.222286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:41.222256 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" Oct 30 22:46:41.222463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:41.222284 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r" event={"ID":"0bd552b3-a653-451b-b473-3c8a972ef789","Type":"ContainerDied","Data":"34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3"} Oct 30 22:46:41.222463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:46:41.222332 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e9328e281509d45ed05d2451b177d731e6848d956ee8f6e85dce0d0a03caf3" Oct 30 22:46:56.639107 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:46:56.639066 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:47:01.340364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.340328 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x"] Oct 30 22:47:01.340889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.340870 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0bd552b3-a653-451b-b473-3c8a972ef789" containerName="revoke-keys" Oct 30 22:47:01.340889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.340890 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd552b3-a653-451b-b473-3c8a972ef789" containerName="revoke-keys" Oct 30 22:47:01.340979 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.340954 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd552b3-a653-451b-b473-3c8a972ef789" containerName="revoke-keys" Oct 30 22:47:01.344191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.344176 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.404018 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.403988 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x"] Oct 30 22:47:01.407885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.407856 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4t6\" (UniqueName: \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.408037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.407989 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.408037 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.408036 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.451490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.451456 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:47:01.451490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.451468 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:47:01.451490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.451468 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:47:01.451720 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.451524 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:47:01.509531 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.509492 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.509730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.509635 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4t6\" (UniqueName: \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.509730 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.509707 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.510301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.510277 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.511927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.511900 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.573699 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.573645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4t6\" (UniqueName: \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\") pod \"maas-api-revoke-sub-0b7e8edb-2gr4x\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:01.656957 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:01.656876 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:02.039451 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:02.039423 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x"] Oct 30 22:47:02.041006 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:47:02.040981 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebcb4cb_5a73_46b7_ac70_de79402b6fef.slice/crio-37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a WatchSource:0}: Error finding container 37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a: Status 404 returned error can't find the container with id 37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a Oct 30 22:47:02.326051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:02.326022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" event={"ID":"1ebcb4cb-5a73-46b7-ac70-de79402b6fef","Type":"ContainerStarted","Data":"848fe17737400e6f2ec70aa358c7164f2609c072fe1bf859c1f86af217d620de"} Oct 30 22:47:02.326179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:02.326071 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" event={"ID":"1ebcb4cb-5a73-46b7-ac70-de79402b6fef","Type":"ContainerStarted","Data":"37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a"} Oct 30 22:47:02.454663 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:02.454603 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" podStartSLOduration=1.454588469 podStartE2EDuration="1.454588469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:02.452145227 +0000 UTC m=+3150.420964794" watchObservedRunningTime="2026-10-30 22:47:02.454588469 +0000 UTC m=+3150.423408026" Oct 30 22:47:03.331677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:03.331642 4292 generic.go:356] "Generic (PLEG): container finished" podID="1ebcb4cb-5a73-46b7-ac70-de79402b6fef" containerID="848fe17737400e6f2ec70aa358c7164f2609c072fe1bf859c1f86af217d620de" exitCode=0 Oct 30 22:47:03.331871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:03.331733 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" event={"ID":"1ebcb4cb-5a73-46b7-ac70-de79402b6fef","Type":"ContainerDied","Data":"848fe17737400e6f2ec70aa358c7164f2609c072fe1bf859c1f86af217d620de"} Oct 30 22:47:04.469995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.469972 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:04.538281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.538252 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\") pod \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " Oct 30 22:47:04.538433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.538321 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\") pod \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " Oct 30 22:47:04.538433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.538414 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\" (UniqueName: \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\") pod \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\" (UID: \"1ebcb4cb-5a73-46b7-ac70-de79402b6fef\") " Oct 30 22:47:04.538665 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.538556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca" pod "1ebcb4cb-5a73-46b7-ac70-de79402b6fef" (UID: "1ebcb4cb-5a73-46b7-ac70-de79402b6fef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:04.538752 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.538734 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:04.540337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.540313 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca" pod "1ebcb4cb-5a73-46b7-ac70-de79402b6fef" (UID: "1ebcb4cb-5a73-46b7-ac70-de79402b6fef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:04.540443 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.540370 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6" pod "1ebcb4cb-5a73-46b7-ac70-de79402b6fef" (UID: "1ebcb4cb-5a73-46b7-ac70-de79402b6fef"). InnerVolumeSpecName "kube-api-access-tn4t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:04.640260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.640195 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:04.640260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:04.640218 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4t6\" (UniqueName: \"kubernetes.io/projected/1ebcb4cb-5a73-46b7-ac70-de79402b6fef-kube-api-access-tn4t6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:05.342627 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:05.342596 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" Oct 30 22:47:05.342832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:05.342601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x" event={"ID":"1ebcb4cb-5a73-46b7-ac70-de79402b6fef","Type":"ContainerDied","Data":"37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a"} Oct 30 22:47:05.342832 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:05.342715 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bdd19ba589b8e6d052b626973c2f8139841109f4be5376629421cd1673c87a" Oct 30 22:47:12.825590 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.825542 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq"] Oct 30 22:47:12.827388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.827360 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1ebcb4cb-5a73-46b7-ac70-de79402b6fef" containerName="revoke-keys" Oct 30 22:47:12.827567 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.827552 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebcb4cb-5a73-46b7-ac70-de79402b6fef" containerName="revoke-keys" Oct 30 22:47:12.827921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.827904 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebcb4cb-5a73-46b7-ac70-de79402b6fef" containerName="revoke-keys" Oct 30 22:47:12.832361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.832343 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:12.886453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.886425 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq"] Oct 30 22:47:12.920769 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.920736 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d472\" (UniqueName: \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:12.920908 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.920806 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:12.920951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.920913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:12.933810 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.933782 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:47:12.933918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.933824 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:47:12.933918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.933841 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:47:12.936954 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:12.936941 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:47:13.022282 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.022255 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d472\" (UniqueName: \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.022402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.022292 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.022402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.022350 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.023103 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.023075 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.024721 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.024700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.085970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.085917 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d472\" (UniqueName: \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\") pod \"maas-api-revoke-sub-3d4156f1-km4sq\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.145300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.145274 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:13.520454 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:13.520421 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq"] Oct 30 22:47:13.521340 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:47:13.521303 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f46030_45cd_411b_be26_8a7ac85a1a53.slice/crio-24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b WatchSource:0}: Error finding container 24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b: Status 404 returned error can't find the container with id 24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b Oct 30 22:47:14.387582 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:14.387547 4292 generic.go:356] "Generic (PLEG): container finished" podID="38f46030-45cd-411b-be26-8a7ac85a1a53" containerID="28b90a2628cbd465c5688d94b11e872e321b429f0cb647e0e295adc1644fa14b" exitCode=0 Oct 30 22:47:14.387926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:14.387608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" event={"ID":"38f46030-45cd-411b-be26-8a7ac85a1a53","Type":"ContainerDied","Data":"28b90a2628cbd465c5688d94b11e872e321b429f0cb647e0e295adc1644fa14b"} Oct 30 22:47:14.387926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:14.387680 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" event={"ID":"38f46030-45cd-411b-be26-8a7ac85a1a53","Type":"ContainerStarted","Data":"24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b"} Oct 30 22:47:15.765859 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.765533 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:15.872539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.872507 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\") pod \"38f46030-45cd-411b-be26-8a7ac85a1a53\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " Oct 30 22:47:15.872767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.872639 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\" (UniqueName: \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\") pod \"38f46030-45cd-411b-be26-8a7ac85a1a53\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " Oct 30 22:47:15.872855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.872827 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\") pod \"38f46030-45cd-411b-be26-8a7ac85a1a53\" (UID: \"38f46030-45cd-411b-be26-8a7ac85a1a53\") " Oct 30 22:47:15.873275 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.873173 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca" pod "38f46030-45cd-411b-be26-8a7ac85a1a53" (UID: "38f46030-45cd-411b-be26-8a7ac85a1a53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:15.873410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.873295 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:15.874793 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.874766 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca" pod "38f46030-45cd-411b-be26-8a7ac85a1a53" (UID: "38f46030-45cd-411b-be26-8a7ac85a1a53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:15.875321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.875301 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472" pod "38f46030-45cd-411b-be26-8a7ac85a1a53" (UID: "38f46030-45cd-411b-be26-8a7ac85a1a53"). InnerVolumeSpecName "kube-api-access-7d472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:15.974520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.974439 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f46030-45cd-411b-be26-8a7ac85a1a53-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:15.974520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:15.974471 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d472\" (UniqueName: \"kubernetes.io/projected/38f46030-45cd-411b-be26-8a7ac85a1a53-kube-api-access-7d472\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:16.410679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:16.410641 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" event={"ID":"38f46030-45cd-411b-be26-8a7ac85a1a53","Type":"ContainerDied","Data":"24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b"} Oct 30 22:47:16.410890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:16.410704 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24569f6ee6cb03339882e8c6dd85bc0177c7f358c14bf37aa6646b9a5b10d33b" Oct 30 22:47:16.410890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:16.410658 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq" Oct 30 22:47:23.057025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:23.056990 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7"] Oct 30 22:47:23.107336 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:23.107304 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fxzx7"] Oct 30 22:47:24.145253 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:24.145216 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72aa6e74-5fcd-478f-a36f-871a0672a615" path="/var/lib/kubelet/pods/72aa6e74-5fcd-478f-a36f-871a0672a615/volumes" Oct 30 22:47:26.637808 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:47:26.637768 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:47:38.190362 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:38.190330 4292 scope.go:122] "RemoveContainer" containerID="a220187312e683f7334d12f37947edbee5098e548acb8d728a7723fed3a33566" Oct 30 22:47:46.670324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.670243 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq"] Oct 30 22:47:46.721193 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.721161 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-km4sq"] Oct 30 22:47:46.795406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.795378 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs"] Oct 30 22:47:46.795804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.795787 4292 cpu_manager.go:414] "containerMap: removing container" podUID="38f46030-45cd-411b-be26-8a7ac85a1a53" containerName="revoke-keys" Oct 30 22:47:46.795804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.795803 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f46030-45cd-411b-be26-8a7ac85a1a53" containerName="revoke-keys" Oct 30 22:47:46.795958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.795900 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f46030-45cd-411b-be26-8a7ac85a1a53" containerName="revoke-keys" Oct 30 22:47:46.800219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.800200 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:46.857128 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.857096 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:47:46.857280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.857129 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:47:46.857280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.857229 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:47:46.857280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.857236 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:47:46.858712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.858688 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs"] Oct 30 22:47:46.934315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.934222 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xcs\" (UniqueName: \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:46.934315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.934283 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:46.934551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:46.934394 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.035370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.035343 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.035511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.035420 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xcs\" (UniqueName: \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.035511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.035455 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.036216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.036191 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.037822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.037798 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.099545 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.099517 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xcs\" (UniqueName: \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\") pod \"maas-api-revoke-sub-3d4156f1-t4xrs\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.112281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.112263 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:47.489703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.489673 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs"] Oct 30 22:47:47.491066 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:47:47.491038 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda3eb92_1a35_4ba5_9f0c_ec588c5ffd3d.slice/crio-49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214 WatchSource:0}: Error finding container 49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214: Status 404 returned error can't find the container with id 49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214 Oct 30 22:47:47.609277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:47.609243 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" event={"ID":"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d","Type":"ContainerStarted","Data":"49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214"} Oct 30 22:47:48.144885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:48.144850 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f46030-45cd-411b-be26-8a7ac85a1a53" path="/var/lib/kubelet/pods/38f46030-45cd-411b-be26-8a7ac85a1a53/volumes" Oct 30 22:47:48.616610 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:48.616553 4292 generic.go:356] "Generic (PLEG): container finished" podID="fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" containerID="942f4e9080a024004317202fb42fbe585ba8d1534938d94bda7ac66fc47c3a85" exitCode=0 Oct 30 22:47:48.616799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:48.616635 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" event={"ID":"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d","Type":"ContainerDied","Data":"942f4e9080a024004317202fb42fbe585ba8d1534938d94bda7ac66fc47c3a85"} Oct 30 22:47:49.751505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.751480 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:49.862697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.862669 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\") pod \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " Oct 30 22:47:49.862839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.862748 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\" (UniqueName: \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\") pod \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " Oct 30 22:47:49.862839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.862793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\") pod \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\" (UID: \"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d\") " Oct 30 22:47:49.863206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.863180 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca" pod "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" (UID: "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:49.864767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.864745 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs" pod "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" (UID: "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d"). InnerVolumeSpecName "kube-api-access-b9xcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:49.864853 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.864799 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca" pod "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" (UID: "fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:49.963946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.963861 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.963946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.963893 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.963946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:49.963909 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xcs\" (UniqueName: \"kubernetes.io/projected/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d-kube-api-access-b9xcs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:47:50.628599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:50.628552 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" Oct 30 22:47:50.628817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:50.628602 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs" event={"ID":"fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d","Type":"ContainerDied","Data":"49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214"} Oct 30 22:47:50.628817 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:47:50.628662 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a664587a7c992a45e79053853b8ee084a519d563307ab9fbb68d58051b0214" Oct 30 22:47:56.638427 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:47:56.638390 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:48:26.639071 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:48:26.639029 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:48:30.412729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:30.412685 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:48:30.413223 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:30.412760 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:48:34.485279 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.485247 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt"] Oct 30 22:48:34.485683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.485668 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" containerName="revoke-keys" Oct 30 22:48:34.485740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.485683 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" containerName="revoke-keys" Oct 30 22:48:34.485787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.485776 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" containerName="revoke-keys" Oct 30 22:48:34.489067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.489051 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.544086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.544065 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:48:34.544086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.544067 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:48:34.594000 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.593965 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:48:34.594146 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.594016 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:48:34.596243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.596225 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt"] Oct 30 22:48:34.600538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.600519 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.600614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.600563 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.600759 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.600743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlhd\" (UniqueName: \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.701825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.701797 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.701989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.701849 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.701989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.701918 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlhd\" (UniqueName: \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.702564 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.702540 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.704083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.704056 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.765303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.765252 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlhd\" (UniqueName: \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\") pod \"maas-api-revoke-sub-e2fc9c5b-qnbgt\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:34.801457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:34.801436 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:35.176549 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:35.176518 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt"] Oct 30 22:48:35.177834 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:48:35.177806 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1a5e61_81e9_4d76_961d_998923e3f06f.slice/crio-51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97 WatchSource:0}: Error finding container 51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97: Status 404 returned error can't find the container with id 51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97 Oct 30 22:48:35.888277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:35.888234 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" event={"ID":"5d1a5e61-81e9-4d76-961d-998923e3f06f","Type":"ContainerStarted","Data":"b245bc0c62ae823f166cdfe665534e3ac3f0983aa75f85b9971a210d17e809f0"} Oct 30 22:48:35.888713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:35.888304 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" event={"ID":"5d1a5e61-81e9-4d76-961d-998923e3f06f","Type":"ContainerStarted","Data":"51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97"} Oct 30 22:48:36.009869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:36.009818 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" podStartSLOduration=2.009802556 podStartE2EDuration="2.009802556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:36.008242133 +0000 UTC m=+3243.977061699" watchObservedRunningTime="2026-10-30 22:48:36.009802556 +0000 UTC m=+3243.978622163" Oct 30 22:48:36.894229 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:36.894194 4292 generic.go:356] "Generic (PLEG): container finished" podID="5d1a5e61-81e9-4d76-961d-998923e3f06f" containerID="b245bc0c62ae823f166cdfe665534e3ac3f0983aa75f85b9971a210d17e809f0" exitCode=0 Oct 30 22:48:36.894643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:36.894235 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" event={"ID":"5d1a5e61-81e9-4d76-961d-998923e3f06f","Type":"ContainerDied","Data":"b245bc0c62ae823f166cdfe665534e3ac3f0983aa75f85b9971a210d17e809f0"} Oct 30 22:48:38.031703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.031680 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:38.129208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.129179 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\") pod \"5d1a5e61-81e9-4d76-961d-998923e3f06f\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " Oct 30 22:48:38.129325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.129215 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\") pod \"5d1a5e61-81e9-4d76-961d-998923e3f06f\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " Oct 30 22:48:38.129325 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.129303 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\" (UniqueName: \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\") pod \"5d1a5e61-81e9-4d76-961d-998923e3f06f\" (UID: \"5d1a5e61-81e9-4d76-961d-998923e3f06f\") " Oct 30 22:48:38.129633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.129604 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca" pod "5d1a5e61-81e9-4d76-961d-998923e3f06f" (UID: "5d1a5e61-81e9-4d76-961d-998923e3f06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:38.131233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.131196 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca" pod "5d1a5e61-81e9-4d76-961d-998923e3f06f" (UID: "5d1a5e61-81e9-4d76-961d-998923e3f06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:38.131357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.131340 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd" pod "5d1a5e61-81e9-4d76-961d-998923e3f06f" (UID: "5d1a5e61-81e9-4d76-961d-998923e3f06f"). InnerVolumeSpecName "kube-api-access-5qlhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:38.215200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.215176 4292 scope.go:122] "RemoveContainer" containerID="ddc2db18307700d8ed47d1ef74eda8aa3fbbfe5a7564d3edc017c14b1c3a9b83" Oct 30 22:48:38.229946 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.229924 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qlhd\" (UniqueName: \"kubernetes.io/projected/5d1a5e61-81e9-4d76-961d-998923e3f06f-kube-api-access-5qlhd\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:38.230038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.229950 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:38.230038 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.229967 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1a5e61-81e9-4d76-961d-998923e3f06f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:38.905751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.905718 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" Oct 30 22:48:38.905751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.905730 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt" event={"ID":"5d1a5e61-81e9-4d76-961d-998923e3f06f","Type":"ContainerDied","Data":"51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97"} Oct 30 22:48:38.905960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:38.905777 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b68201cf6f85dfc604ccfac20abe0d6136f8c7f12a39cc415b48515615cd97" Oct 30 22:48:45.258872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.258833 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv"] Oct 30 22:48:45.259514 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.259491 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5d1a5e61-81e9-4d76-961d-998923e3f06f" containerName="revoke-keys" Oct 30 22:48:45.259588 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.259520 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1a5e61-81e9-4d76-961d-998923e3f06f" containerName="revoke-keys" Oct 30 22:48:45.259671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.259657 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1a5e61-81e9-4d76-961d-998923e3f06f" containerName="revoke-keys" Oct 30 22:48:45.264176 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.264156 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.325273 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.325241 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv"] Oct 30 22:48:45.372532 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.372501 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:48:45.372809 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.372792 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:48:45.372900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.372817 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:48:45.372900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.372835 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:48:45.396111 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.396089 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz8c\" (UniqueName: \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.396233 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.396214 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.396289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.396250 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.497337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.497311 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz8c\" (UniqueName: \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.497469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.497400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.497469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.497441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.498082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.498060 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.499675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.499656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.572693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.572673 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz8c\" (UniqueName: \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\") pod \"maas-api-revoke-sub-5a29efb3-92ctv\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.576481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.576461 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:45.950620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:45.950592 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv"] Oct 30 22:48:45.952290 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:48:45.952259 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e78d1c_be93_4b87_a2e1_a666a5c5c9d1.slice/crio-82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60 WatchSource:0}: Error finding container 82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60: Status 404 returned error can't find the container with id 82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60 Oct 30 22:48:46.945711 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:46.945678 4292 generic.go:356] "Generic (PLEG): container finished" podID="e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" containerID="12035c416c67df9e40a299ef436fa92b593b4317868541d379f51ae76b5d45e0" exitCode=0 Oct 30 22:48:46.946095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:46.945776 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" event={"ID":"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1","Type":"ContainerDied","Data":"12035c416c67df9e40a299ef436fa92b593b4317868541d379f51ae76b5d45e0"} Oct 30 22:48:46.946095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:46.945827 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" event={"ID":"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1","Type":"ContainerStarted","Data":"82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60"} Oct 30 22:48:48.108911 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.108880 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:48.223999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.223917 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\" (UniqueName: \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\") pod \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " Oct 30 22:48:48.223999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.223978 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\") pod \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " Oct 30 22:48:48.224214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.224039 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\") pod \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\" (UID: \"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1\") " Oct 30 22:48:48.224466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.224432 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca" pod "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" (UID: "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:48.226167 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.226133 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca" pod "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" (UID: "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:48.226570 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.226547 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c" pod "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" (UID: "e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1"). InnerVolumeSpecName "kube-api-access-tpz8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:48.325991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.325938 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz8c\" (UniqueName: \"kubernetes.io/projected/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-kube-api-access-tpz8c\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.325991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.325984 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.326250 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.326001 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.962326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.962282 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" event={"ID":"e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1","Type":"ContainerDied","Data":"82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60"} Oct 30 22:48:48.962556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.962351 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82fdda53d051aece7c7f7597c3efde1d39ba9caee5963fba61efefff61679a60" Oct 30 22:48:48.962556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:48:48.962427 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv" Oct 30 22:48:56.639144 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:48:56.639100 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:49:02.059115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.059072 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp"] Oct 30 22:49:02.109551 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.109513 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zlqbp"] Oct 30 22:49:02.146260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.146222 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79d38aa-ff5f-427c-b8c4-d22d71130475" path="/var/lib/kubelet/pods/e79d38aa-ff5f-427c-b8c4-d22d71130475/volumes" Oct 30 22:49:02.739019 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.738982 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs"] Oct 30 22:49:02.789552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.789524 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4xrs"] Oct 30 22:49:02.859921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.859879 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np"] Oct 30 22:49:02.860324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.860310 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" containerName="revoke-keys" Oct 30 22:49:02.860386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.860326 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" containerName="revoke-keys" Oct 30 22:49:02.860422 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.860415 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" containerName="revoke-keys" Oct 30 22:49:02.863869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.863853 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:02.918370 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.918337 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:49:02.918536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.918418 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:49:02.918536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.918494 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:49:02.918536 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.918510 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:49:02.923614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.923588 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np"] Oct 30 22:49:02.978684 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.978649 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:02.978858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.978813 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrzd\" (UniqueName: \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:02.978858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:02.978844 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.079872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.079845 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrzd\" (UniqueName: \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.080284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.079877 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.080284 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.079921 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.080614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.080564 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.082240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.082211 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.144098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.144070 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrzd\" (UniqueName: \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\") pod \"maas-api-revoke-sub-3d4156f1-f98np\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.176910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.176878 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:03.555892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:03.555852 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np"] Oct 30 22:49:03.557178 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:49:03.557150 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726bd272_46d5_424a_b30d_56b92e0f0dcc.slice/crio-5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7 WatchSource:0}: Error finding container 5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7: Status 404 returned error can't find the container with id 5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7 Oct 30 22:49:04.037110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:04.037078 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" event={"ID":"726bd272-46d5-424a-b30d-56b92e0f0dcc","Type":"ContainerStarted","Data":"ed594165bb796eeae83efa35971217297d2ddef0eaf689500799667f76a84d8d"} Oct 30 22:49:04.037267 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:04.037131 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" event={"ID":"726bd272-46d5-424a-b30d-56b92e0f0dcc","Type":"ContainerStarted","Data":"5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7"} Oct 30 22:49:04.145181 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:04.145109 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d" path="/var/lib/kubelet/pods/fda3eb92-1a35-4ba5-9f0c-ec588c5ffd3d/volumes" Oct 30 22:49:05.043500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:05.043466 4292 generic.go:356] "Generic (PLEG): container finished" podID="726bd272-46d5-424a-b30d-56b92e0f0dcc" containerID="ed594165bb796eeae83efa35971217297d2ddef0eaf689500799667f76a84d8d" exitCode=0 Oct 30 22:49:05.043738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:05.043554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" event={"ID":"726bd272-46d5-424a-b30d-56b92e0f0dcc","Type":"ContainerDied","Data":"ed594165bb796eeae83efa35971217297d2ddef0eaf689500799667f76a84d8d"} Oct 30 22:49:06.188548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.188521 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:06.316236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.316155 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\") pod \"726bd272-46d5-424a-b30d-56b92e0f0dcc\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " Oct 30 22:49:06.316236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.316208 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\" (UniqueName: \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\") pod \"726bd272-46d5-424a-b30d-56b92e0f0dcc\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " Oct 30 22:49:06.316471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.316395 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\") pod \"726bd272-46d5-424a-b30d-56b92e0f0dcc\" (UID: \"726bd272-46d5-424a-b30d-56b92e0f0dcc\") " Oct 30 22:49:06.316799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.316625 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca" pod "726bd272-46d5-424a-b30d-56b92e0f0dcc" (UID: "726bd272-46d5-424a-b30d-56b92e0f0dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:06.316799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.316769 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.318943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.318919 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca" pod "726bd272-46d5-424a-b30d-56b92e0f0dcc" (UID: "726bd272-46d5-424a-b30d-56b92e0f0dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:06.319056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.318953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd" pod "726bd272-46d5-424a-b30d-56b92e0f0dcc" (UID: "726bd272-46d5-424a-b30d-56b92e0f0dcc"). InnerVolumeSpecName "kube-api-access-xqrzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:06.417856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.417819 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726bd272-46d5-424a-b30d-56b92e0f0dcc-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.417856 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:06.417851 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqrzd\" (UniqueName: \"kubernetes.io/projected/726bd272-46d5-424a-b30d-56b92e0f0dcc-kube-api-access-xqrzd\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:07.055943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:07.055910 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" Oct 30 22:49:07.056124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:07.055942 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np" event={"ID":"726bd272-46d5-424a-b30d-56b92e0f0dcc","Type":"ContainerDied","Data":"5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7"} Oct 30 22:49:07.056124 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:07.055993 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe13adc68d923e4dc3037806fba6a206b5273b6cab5f3a5e874415ffe9ff6c7" Oct 30 22:49:14.079206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:14.079170 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp"] Oct 30 22:49:14.130677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:14.130641 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ftlcp"] Oct 30 22:49:14.145642 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:14.145613 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408f0fe1-b7ca-451f-a5f3-4cb59e989b40" path="/var/lib/kubelet/pods/408f0fe1-b7ca-451f-a5f3-4cb59e989b40/volumes" Oct 30 22:49:26.638472 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:49:26.638433 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:49:35.399982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.399951 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:49:35.400506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.400346 4292 cpu_manager.go:414] "containerMap: removing container" podUID="726bd272-46d5-424a-b30d-56b92e0f0dcc" containerName="revoke-keys" Oct 30 22:49:35.400506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.400357 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726bd272-46d5-424a-b30d-56b92e0f0dcc" containerName="revoke-keys" Oct 30 22:49:35.400506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.400464 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="726bd272-46d5-424a-b30d-56b92e0f0dcc" containerName="revoke-keys" Oct 30 22:49:35.404728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.404711 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.460818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.460779 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-72648e-openshift-default-dockercfg-f4hcg\"" Oct 30 22:49:35.503364 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503334 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzftb\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503370 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503387 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503535 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503414 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503559 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503613 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503643 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503752 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.503808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.503774 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.512949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.512919 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:49:35.604317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzftb\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604319 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604342 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604373 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604439 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604490 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604550 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604530 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604611 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604824 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604646 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.604952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604888 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.605017 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.604959 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.605079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.605055 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.605132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.605096 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.605171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.605145 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.606939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.606915 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.607104 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.607090 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.669638 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.669558 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.670376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.670351 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzftb\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\") pod \"e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:35.717430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:35.717410 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:36.146349 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:49:36.146316 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc7f7a8_5fd1_4690_a8f7_fdebd02af4a9.slice/crio-7fe8204bf8b841d2fff8fc3625cdc5525450609149790b366d4fa680690be6b5 WatchSource:0}: Error finding container 7fe8204bf8b841d2fff8fc3625cdc5525450609149790b366d4fa680690be6b5: Status 404 returned error can't find the container with id 7fe8204bf8b841d2fff8fc3625cdc5525450609149790b366d4fa680690be6b5 Oct 30 22:49:36.147496 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.147471 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:49:36.148400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.148374 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:36.148464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.148452 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:36.148505 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.148482 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:36.230608 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.230565 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerStarted","Data":"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff"} Oct 30 22:49:36.230756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.230629 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerStarted","Data":"7fe8204bf8b841d2fff8fc3625cdc5525450609149790b366d4fa680690be6b5"} Oct 30 22:49:36.357020 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.356970 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podStartSLOduration=1.356957831 podStartE2EDuration="1.356957831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:36.356023442 +0000 UTC m=+3304.324843007" watchObservedRunningTime="2026-10-30 22:49:36.356957831 +0000 UTC m=+3304.325777395" Oct 30 22:49:36.717912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:36.717878 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:37.723420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:37.723383 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:38.240552 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:38.240522 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:38.241321 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:38.241296 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:49:43.095609 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.095564 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:49:43.099352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.099336 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.154260 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.154240 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-72648e\"" Oct 30 22:49:43.167865 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.167840 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvf2\" (UniqueName: \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.168027 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.167881 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.203821 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.203783 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fac382a5-dockercfg-k58zl\"" Oct 30 22:49:43.206390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.206364 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:49:43.258272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.258244 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:49:43.261812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.261793 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.268492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.268472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvf2\" (UniqueName: \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.268594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.268518 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.269402 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.269378 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.289252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.289228 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:49:43.331777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.331754 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvf2\" (UniqueName: \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\") pod \"payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.369761 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.369711 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.369864 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.369777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlv5r\" (UniqueName: \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.412256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.412234 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:43.470709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.470678 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlv5r\" (UniqueName: \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.471481 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.471381 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.472187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.472168 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.534981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.534955 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlv5r\" (UniqueName: \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\") pod \"payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.574938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.574911 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:43.787251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.787221 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:49:43.788125 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:49:43.788098 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d54c09_5e7d_4bff_a029_875ee3ab8d40.slice/crio-303eaa4740998d3aa677260a127c6d56ad8ffbb9fa76dbdac1a4ad22dfe25178 WatchSource:0}: Error finding container 303eaa4740998d3aa677260a127c6d56ad8ffbb9fa76dbdac1a4ad22dfe25178: Status 404 returned error can't find the container with id 303eaa4740998d3aa677260a127c6d56ad8ffbb9fa76dbdac1a4ad22dfe25178 Oct 30 22:49:43.948203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:43.948178 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:49:43.949996 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:49:43.949967 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4bd886_94a8_4a0a_ac4e_b97f53eb01b4.slice/crio-2c1cba71c4052a1680dc4874f73db3695769c70c2eec86ee7b11b0c6e6cd6ba3 WatchSource:0}: Error finding container 2c1cba71c4052a1680dc4874f73db3695769c70c2eec86ee7b11b0c6e6cd6ba3: Status 404 returned error can't find the container with id 2c1cba71c4052a1680dc4874f73db3695769c70c2eec86ee7b11b0c6e6cd6ba3 Oct 30 22:49:44.270345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.270261 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" event={"ID":"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4","Type":"ContainerStarted","Data":"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8"} Oct 30 22:49:44.270345 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.270316 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" event={"ID":"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4","Type":"ContainerStarted","Data":"2c1cba71c4052a1680dc4874f73db3695769c70c2eec86ee7b11b0c6e6cd6ba3"} Oct 30 22:49:44.270861 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.270372 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:44.271917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.271881 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" event={"ID":"e5d54c09-5e7d-4bff-a029-875ee3ab8d40","Type":"ContainerStarted","Data":"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34"} Oct 30 22:49:44.272045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.271926 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" event={"ID":"e5d54c09-5e7d-4bff-a029-875ee3ab8d40","Type":"ContainerStarted","Data":"303eaa4740998d3aa677260a127c6d56ad8ffbb9fa76dbdac1a4ad22dfe25178"} Oct 30 22:49:44.272045 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.271953 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:44.391644 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.391592 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" podStartSLOduration=1.39156013 podStartE2EDuration="1.39156013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:44.390341179 +0000 UTC m=+3312.359160744" watchObservedRunningTime="2026-10-30 22:49:44.39156013 +0000 UTC m=+3312.360379695" Oct 30 22:49:44.517365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:44.517312 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" podStartSLOduration=1.517297954 podStartE2EDuration="1.517297954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:44.517101182 +0000 UTC m=+3312.485920746" watchObservedRunningTime="2026-10-30 22:49:44.517297954 +0000 UTC m=+3312.486117519" Oct 30 22:49:50.056697 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:50.056663 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl"] Oct 30 22:49:50.107123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:50.107092 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-69bbl"] Oct 30 22:49:50.145212 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:50.145184 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5bf0a2-7fe4-4eb2-8959-186e79f40553" path="/var/lib/kubelet/pods/3f5bf0a2-7fe4-4eb2-8959-186e79f40553/volumes" Oct 30 22:49:55.279376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:55.279343 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:49:55.279825 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:49:55.279423 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:49:56.639545 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:49:56.639509 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:50:00.058466 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:00.058432 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk"] Oct 30 22:50:00.060447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:00.060417 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4m9lk"] Oct 30 22:50:00.146367 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:00.146326 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf762fb2-587e-4d2c-a8b0-81ef21c3201a" path="/var/lib/kubelet/pods/bf762fb2-587e-4d2c-a8b0-81ef21c3201a/volumes" Oct 30 22:50:03.366432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.366338 4292 generic.go:356] "Generic (PLEG): container finished" podID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerID="e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff" exitCode=0 Oct 30 22:50:03.366432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.366415 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerDied","Data":"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff"} Oct 30 22:50:03.366924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.366860 4292 scope.go:122] "RemoveContainer" containerID="e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff" Oct 30 22:50:03.368161 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.368134 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:50:03.368241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.368199 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:50:03.368241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:03.368236 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:50:04.374005 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:04.373972 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerStarted","Data":"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d"} Oct 30 22:50:04.718297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:04.718198 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:50:04.719493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:04.719464 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 22:50:04.719631 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:04.719525 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 22:50:05.718237 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:05.718200 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 22:50:05.718708 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:05.718479 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 22:50:06.718129 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:06.718093 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 22:50:06.718406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:06.718160 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 22:50:07.718272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.718235 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 22:50:07.718457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.718283 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 22:50:07.905599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.901157 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:50:07.913839 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.913815 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.946724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.946696 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.946899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.946885 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.947123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.947094 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfwl\" (UniqueName: \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.947192 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.947172 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.947299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.947245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.947365 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.947313 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:07.958653 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:07.957257 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:50:08.009792 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.009710 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-72648e\"/\"kube-root-ca.crt\"" Oct 30 22:50:08.010012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.009719 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-72648e\"/\"openshift-service-ca.crt\"" Oct 30 22:50:08.010113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.009851 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-72648e\"/\"default-dockercfg-29d8t\"" Oct 30 22:50:08.018628 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.018612 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-72648e\"/\"simulator-single-main-72648e-kserve-self-signed-certs\"" Oct 30 22:50:08.048894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.048864 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfwl\" (UniqueName: \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049092 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049203 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049292 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049386 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049538 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049510 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.049789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.049766 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.050116 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.050095 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.050281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.050254 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.052390 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.052368 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.053263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.053243 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.118945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.118913 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfwl\" (UniqueName: \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\") pod \"simulator-single-main-72648e-kserve-7b999cc554-w6nn4\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.226348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.226320 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:08.241528 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.241504 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:50:08.620600 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.620557 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:50:08.621527 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:50:08.621490 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20955bf4_a275_402e_9311_69834ea66fc2.slice/crio-104152f32e31de341ae2ef8c156178cc85779e726bb41a6d44e62210dc96659c WatchSource:0}: Error finding container 104152f32e31de341ae2ef8c156178cc85779e726bb41a6d44e62210dc96659c: Status 404 returned error can't find the container with id 104152f32e31de341ae2ef8c156178cc85779e726bb41a6d44e62210dc96659c Oct 30 22:50:08.718148 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.718115 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 22:50:08.718315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:08.718179 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 22:50:09.421797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:09.421760 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" event={"ID":"20955bf4-a275-402e-9311-69834ea66fc2","Type":"ContainerStarted","Data":"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef"} Oct 30 22:50:09.422304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:09.421806 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" event={"ID":"20955bf4-a275-402e-9311-69834ea66fc2","Type":"ContainerStarted","Data":"104152f32e31de341ae2ef8c156178cc85779e726bb41a6d44e62210dc96659c"} Oct 30 22:50:09.543899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:09.543840 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" podStartSLOduration=2.543825052 podStartE2EDuration="2.543825052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:09.543356353 +0000 UTC m=+3337.512176017" watchObservedRunningTime="2026-10-30 22:50:09.543825052 +0000 UTC m=+3337.512644614" Oct 30 22:50:09.722067 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:09.721982 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:50:09.723183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:09.723162 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:50:10.226885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:10.226853 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:18.226916 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:18.226878 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:18.239240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:18.239217 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:18.484337 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:18.484275 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:50:26.638905 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:50:26.638867 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:50:38.243376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:38.243345 4292 scope.go:122] "RemoveContainer" containerID="e2c123aa4139344151ff9b9507bdb0f46b2994602691cfd9bc9557b1b5343cef" Oct 30 22:50:38.252433 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:38.252411 4292 scope.go:122] "RemoveContainer" containerID="6b5d1c9117d7e197cead7ff69da52115adacab1fc61a027a2e76424c7649cbd2" Oct 30 22:50:39.055654 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:39.055621 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd"] Oct 30 22:50:39.105900 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:39.105872 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xlrhd"] Oct 30 22:50:40.145656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:40.145621 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469f9b88-2098-485e-b0de-e31d3f8f84cf" path="/var/lib/kubelet/pods/469f9b88-2098-485e-b0de-e31d3f8f84cf/volumes" Oct 30 22:50:56.639513 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:50:56.639431 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:50:58.030452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.030420 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr"] Oct 30 22:50:58.035418 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.035397 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.092234 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.092203 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.092383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.092257 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnpm\" (UniqueName: \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.092383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.092358 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.094682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.094655 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr"] Oct 30 22:50:58.142639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.142610 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:50:58.142639 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.142617 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:50:58.142838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.142645 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:50:58.142838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.142745 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:50:58.193675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.193648 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.193807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.193700 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnpm\" (UniqueName: \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.193807 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.193727 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.194615 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.194569 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.196076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.196054 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.258230 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.258205 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnpm\" (UniqueName: \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\") pod \"maas-api-revoke-sub-03dabdfd-dvkhr\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.350904 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.350883 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:50:58.724530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:58.724506 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr"] Oct 30 22:50:58.726277 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:50:58.726249 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b7bf78_5d77_4293_a203_be21692a3ba2.slice/crio-b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c WatchSource:0}: Error finding container b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c: Status 404 returned error can't find the container with id b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c Oct 30 22:50:59.683408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:59.683379 4292 generic.go:356] "Generic (PLEG): container finished" podID="06b7bf78-5d77-4293-a203-be21692a3ba2" containerID="036f6164b6664cb24078199a153ae251c3f0c7e39c4951cdb5c33aa1c151fa09" exitCode=0 Oct 30 22:50:59.683966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:59.683438 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" event={"ID":"06b7bf78-5d77-4293-a203-be21692a3ba2","Type":"ContainerDied","Data":"036f6164b6664cb24078199a153ae251c3f0c7e39c4951cdb5c33aa1c151fa09"} Oct 30 22:50:59.683966 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:50:59.683475 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" event={"ID":"06b7bf78-5d77-4293-a203-be21692a3ba2","Type":"ContainerStarted","Data":"b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c"} Oct 30 22:51:00.819931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.819903 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:51:00.920915 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.920886 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\" (UniqueName: \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\") pod \"06b7bf78-5d77-4293-a203-be21692a3ba2\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " Oct 30 22:51:00.921086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.920960 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\") pod \"06b7bf78-5d77-4293-a203-be21692a3ba2\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " Oct 30 22:51:00.921086 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.920979 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\") pod \"06b7bf78-5d77-4293-a203-be21692a3ba2\" (UID: \"06b7bf78-5d77-4293-a203-be21692a3ba2\") " Oct 30 22:51:00.921330 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.921305 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca" pod "06b7bf78-5d77-4293-a203-be21692a3ba2" (UID: "06b7bf78-5d77-4293-a203-be21692a3ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:00.923113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.923085 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm" pod "06b7bf78-5d77-4293-a203-be21692a3ba2" (UID: "06b7bf78-5d77-4293-a203-be21692a3ba2"). InnerVolumeSpecName "kube-api-access-dbnpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:00.923244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:00.923220 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca" pod "06b7bf78-5d77-4293-a203-be21692a3ba2" (UID: "06b7bf78-5d77-4293-a203-be21692a3ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:01.021852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.021779 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnpm\" (UniqueName: \"kubernetes.io/projected/06b7bf78-5d77-4293-a203-be21692a3ba2-kube-api-access-dbnpm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:01.021852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.021809 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:01.021852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.021822 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b7bf78-5d77-4293-a203-be21692a3ba2-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:01.695077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.695045 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" Oct 30 22:51:01.695077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.695066 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr" event={"ID":"06b7bf78-5d77-4293-a203-be21692a3ba2","Type":"ContainerDied","Data":"b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c"} Oct 30 22:51:01.695341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:01.695117 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b623c1c134ea27bffd368e0b26bba7b941b61a3551697bb55bb71d9648f2861c" Oct 30 22:51:10.341682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.341648 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs"] Oct 30 22:51:10.342184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.342057 4292 cpu_manager.go:414] "containerMap: removing container" podUID="06b7bf78-5d77-4293-a203-be21692a3ba2" containerName="revoke-keys" Oct 30 22:51:10.342184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.342068 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b7bf78-5d77-4293-a203-be21692a3ba2" containerName="revoke-keys" Oct 30 22:51:10.342184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.342150 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b7bf78-5d77-4293-a203-be21692a3ba2" containerName="revoke-keys" Oct 30 22:51:10.345314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.345298 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.404667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.404641 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs"] Oct 30 22:51:10.404989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.404966 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.405060 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.405016 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722fh\" (UniqueName: \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.405186 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.405159 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.452432 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.452405 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:51:10.452556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.452444 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:51:10.452556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.452444 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:51:10.452556 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.452522 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:51:10.506352 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.506329 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.506464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.506365 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722fh\" (UniqueName: \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.506464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.506431 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.506963 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.506943 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.508748 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.508729 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.570042 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.570019 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722fh\" (UniqueName: \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:10.658117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:10.658059 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:11.033872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.033845 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs"] Oct 30 22:51:11.036052 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:51:11.036017 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e8be32_00f7_484c_a6b5_c39f70cdfe6f.slice/crio-5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f WatchSource:0}: Error finding container 5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f: Status 404 returned error can't find the container with id 5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f Oct 30 22:51:11.058065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.058043 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb"] Oct 30 22:51:11.062842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.062823 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkkfb"] Oct 30 22:51:11.757912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.757871 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" event={"ID":"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f","Type":"ContainerStarted","Data":"a7e25a867ef8144ad2b8ca53730ad0205be058414f821d26c422d27254f3c565"} Oct 30 22:51:11.758300 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.757954 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" event={"ID":"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f","Type":"ContainerStarted","Data":"5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f"} Oct 30 22:51:11.877949 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:11.877905 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" podStartSLOduration=1.877891695 podStartE2EDuration="1.877891695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:11.876722851 +0000 UTC m=+3399.845542416" watchObservedRunningTime="2026-10-30 22:51:11.877891695 +0000 UTC m=+3399.846711256" Oct 30 22:51:12.145863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:12.145832 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8486d76-1373-45b6-9671-5474ce6569bb" path="/var/lib/kubelet/pods/e8486d76-1373-45b6-9671-5474ce6569bb/volumes" Oct 30 22:51:12.764973 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:12.764942 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" containerID="a7e25a867ef8144ad2b8ca53730ad0205be058414f821d26c422d27254f3c565" exitCode=0 Oct 30 22:51:12.765363 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:12.765022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" event={"ID":"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f","Type":"ContainerDied","Data":"a7e25a867ef8144ad2b8ca53730ad0205be058414f821d26c422d27254f3c565"} Oct 30 22:51:13.904690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:13.904664 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:14.039882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.039809 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\" (UniqueName: \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\") pod \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " Oct 30 22:51:14.040046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.039893 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\") pod \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " Oct 30 22:51:14.040046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.039969 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\") pod \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\" (UID: \"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f\") " Oct 30 22:51:14.040219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.040197 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca" pod "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" (UID: "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:14.041934 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.041909 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca" pod "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" (UID: "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:14.042076 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.042056 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh" pod "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" (UID: "a0e8be32-00f7-484c-a6b5-c39f70cdfe6f"). InnerVolumeSpecName "kube-api-access-722fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:14.141656 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.141633 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:14.141747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.141657 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722fh\" (UniqueName: \"kubernetes.io/projected/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-kube-api-access-722fh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:14.141747 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.141671 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:14.776518 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.776487 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" Oct 30 22:51:14.776714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.776526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs" event={"ID":"a0e8be32-00f7-484c-a6b5-c39f70cdfe6f","Type":"ContainerDied","Data":"5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f"} Oct 30 22:51:14.776714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.776607 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b36a0d6b7c392b1d4e7be480e6039f18e0d6a9a57cc9aa7f853614bef6e2a0f" Oct 30 22:51:14.804203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:14.804175 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:51:21.840768 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:21.840735 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:51:21.947597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:21.947547 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:51:26.639007 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:26.638972 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:51:26.841138 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:26.841085 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" containerID="cri-o://5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8" gracePeriod=25 Oct 30 22:51:26.947849 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:26.947775 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" containerID="cri-o://2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34" gracePeriod=25 Oct 30 22:51:27.725924 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:27.725873 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs"] Oct 30 22:51:27.775219 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:27.775187 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-72648e-6qtvs"] Oct 30 22:51:28.149187 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:28.149151 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" path="/var/lib/kubelet/pods/a0e8be32-00f7-484c-a6b5-c39f70cdfe6f/volumes" Oct 30 22:51:29.823892 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:29.823839 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" podUID="20955bf4-a275-402e-9311-69834ea66fc2" containerName="main" containerID="cri-o://0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef" gracePeriod=45 Oct 30 22:51:30.080206 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.080138 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:51:30.083378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083348 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.083378 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083379 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.083643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083410 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\" (UniqueName: \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.083643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083435 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.083778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.083835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083783 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.083835 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.083798 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.085482 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.085461 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.085619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.085547 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "kube-api-access-qzfwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:30.184194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184170 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.184328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184209 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\") pod \"20955bf4-a275-402e-9311-69834ea66fc2\" (UID: \"20955bf4-a275-402e-9311-69834ea66fc2\") " Oct 30 22:51:30.184421 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184407 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.184464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184428 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzfwl\" (UniqueName: \"kubernetes.io/projected/20955bf4-a275-402e-9311-69834ea66fc2-kube-api-access-qzfwl\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.184464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184431 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.184464 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.184442 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.186065 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.186046 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs" pod "20955bf4-a275-402e-9311-69834ea66fc2" (UID: "20955bf4-a275-402e-9311-69834ea66fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:30.285446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.285417 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20955bf4-a275-402e-9311-69834ea66fc2-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.285446 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.285444 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20955bf4-a275-402e-9311-69834ea66fc2-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.859679 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.859644 4292 generic.go:356] "Generic (PLEG): container finished" podID="20955bf4-a275-402e-9311-69834ea66fc2" containerID="0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef" exitCode=0 Oct 30 22:51:30.860034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.859687 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" Oct 30 22:51:30.860034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.859687 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" event={"ID":"20955bf4-a275-402e-9311-69834ea66fc2","Type":"ContainerDied","Data":"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef"} Oct 30 22:51:30.860034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.859801 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4" event={"ID":"20955bf4-a275-402e-9311-69834ea66fc2","Type":"ContainerDied","Data":"104152f32e31de341ae2ef8c156178cc85779e726bb41a6d44e62210dc96659c"} Oct 30 22:51:30.860034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.859821 4292 scope.go:122] "RemoveContainer" containerID="0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef" Oct 30 22:51:30.869735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.869717 4292 scope.go:122] "RemoveContainer" containerID="0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef" Oct 30 22:51:30.870031 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:30.870004 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef\": container with ID starting with 0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef not found: ID does not exist" containerID="0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef" Oct 30 22:51:30.870088 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:30.870030 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef"} err="failed to get container status \"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef\": rpc error: code = NotFound desc = could not find container \"0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef\": container with ID starting with 0db4a77ba97673547573e20a34b8acad5f8a9b01cb15f16b7d691d0df9b5a5ef not found: ID does not exist" Oct 30 22:51:31.028840 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:31.028811 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:51:31.079071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:31.079043 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-72648e/simulator-single-main-72648e-kserve-7b999cc554-w6nn4"] Oct 30 22:51:32.144982 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:32.144953 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20955bf4-a275-402e-9311-69834ea66fc2" path="/var/lib/kubelet/pods/20955bf4-a275-402e-9311-69834ea66fc2/volumes" Oct 30 22:51:35.277834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:35.277797 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:35.278277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:35.277853 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:35.555818 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:35.555788 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:51:35.556026 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:35.556001 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" containerID="cri-o://1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d" gracePeriod=30 Oct 30 22:51:37.216266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:37.216229 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np"] Oct 30 22:51:37.266486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:37.266456 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f98np"] Oct 30 22:51:38.145641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:38.145604 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726bd272-46d5-424a-b30d-56b92e0f0dcc" path="/var/lib/kubelet/pods/726bd272-46d5-424a-b30d-56b92e0f0dcc/volumes" Oct 30 22:51:38.278643 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:38.278613 4292 scope.go:122] "RemoveContainer" containerID="85bfc41bc49ff5e418bf459c76f097a14481fa39dd487a7bb0e5c61335500e07" Oct 30 22:51:38.287054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:38.287035 4292 scope.go:122] "RemoveContainer" containerID="2dacc1d92afc8cdac5f4887eab8d3b29d0f71799434abab05928b1464df27c8c" Oct 30 22:51:38.294834 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:38.294815 4292 scope.go:122] "RemoveContainer" containerID="a8c2a9b229d2d4b2bc1c37935268a78ba8e451a949d5c4bcb6733d49bcf4c943" Oct 30 22:51:39.724595 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:39.724552 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:51:39.725046 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:39.724622 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:51:40.808935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.808910 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:51:40.867190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867132 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867190 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867208 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867230 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867273 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867406 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867347 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867428 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867718 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867475 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\") pod \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\" (UID: \"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9\") " Oct 30 22:51:40.867830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867720 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:40.867830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867746 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:40.867830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:40.867830 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867810 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867896 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867966 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867981 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.867996 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.868004 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.868058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.868013 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.869568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.869534 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "kube-api-access-fzftb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:40.869568 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.869594 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:40.869826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.869808 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:51:40.869879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.869855 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy" pod "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" (UID: "5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:40.917372 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.917340 4292 generic.go:356] "Generic (PLEG): container finished" podID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerID="1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d" exitCode=0 Oct 30 22:51:40.917480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.917394 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" Oct 30 22:51:40.917480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.917422 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerDied","Data":"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d"} Oct 30 22:51:40.917480 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.917473 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m" event={"ID":"5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9","Type":"ContainerDied","Data":"7fe8204bf8b841d2fff8fc3625cdc5525450609149790b366d4fa680690be6b5"} Oct 30 22:51:40.917624 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.917496 4292 scope.go:122] "RemoveContainer" containerID="1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d" Oct 30 22:51:40.926605 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.926590 4292 scope.go:122] "RemoveContainer" containerID="e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff" Oct 30 22:51:40.934497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.934480 4292 scope.go:122] "RemoveContainer" containerID="1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d" Oct 30 22:51:40.934758 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:40.934738 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d\": container with ID starting with 1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d not found: ID does not exist" containerID="1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d" Oct 30 22:51:40.934805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.934766 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d"} err="failed to get container status \"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d\": rpc error: code = NotFound desc = could not find container \"1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d\": container with ID starting with 1b35bfbce32d889fb0fff4def3c6d269be84ded1971fd14558a12c82aeb0d87d not found: ID does not exist" Oct 30 22:51:40.934805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.934783 4292 scope.go:122] "RemoveContainer" containerID="e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff" Oct 30 22:51:40.935013 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:40.934994 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff\": container with ID starting with e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff not found: ID does not exist" containerID="e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff" Oct 30 22:51:40.935085 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.935029 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff"} err="failed to get container status \"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff\": rpc error: code = NotFound desc = could not find container \"e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff\": container with ID starting with e49222fa94e0b7a59270f90bfe68e6150f8e8bcb7541d57ac8fddbe0f80144ff not found: ID does not exist" Oct 30 22:51:40.969121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.969095 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.969121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.969119 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.969258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.969135 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzftb\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-kube-api-access-fzftb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:40.969258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:40.969148 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:41.056357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:41.056329 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r"] Oct 30 22:51:41.061437 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:41.061406 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvj7r"] Oct 30 22:51:41.100049 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:41.100024 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:51:41.107880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:41.107855 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-72648e-openshift-default-6cd84bf985shx8m"] Oct 30 22:51:42.145132 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:42.145102 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd552b3-a653-451b-b473-3c8a972ef789" path="/var/lib/kubelet/pods/0bd552b3-a653-451b-b473-3c8a972ef789/volumes" Oct 30 22:51:42.145619 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:42.145600 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" path="/var/lib/kubelet/pods/5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9/volumes" Oct 30 22:51:45.277827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:45.277786 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:45.278235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:45.277864 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:47.084236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.084212 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:51:47.122905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.122873 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\") pod \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " Oct 30 22:51:47.123050 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.122976 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\" (UniqueName: \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\") pod \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\" (UID: \"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4\") " Oct 30 22:51:47.123272 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.123249 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume" pod "ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" (UID: "ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:47.123453 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.123436 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:47.126001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.125972 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r" pod "ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" (UID: "ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4"). InnerVolumeSpecName "kube-api-access-mlv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:47.181121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.181094 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:51:47.224548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.224519 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\") pod \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " Oct 30 22:51:47.224703 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.224646 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\" (UniqueName: \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\") pod \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\" (UID: \"e5d54c09-5e7d-4bff-a029-875ee3ab8d40\") " Oct 30 22:51:47.224846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.224823 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume" pod "e5d54c09-5e7d-4bff-a029-875ee3ab8d40" (UID: "e5d54c09-5e7d-4bff-a029-875ee3ab8d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:47.224888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.224845 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlv5r\" (UniqueName: \"kubernetes.io/projected/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4-kube-api-access-mlv5r\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:47.226726 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.226697 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2" pod "e5d54c09-5e7d-4bff-a029-875ee3ab8d40" (UID: "e5d54c09-5e7d-4bff-a029-875ee3ab8d40"). InnerVolumeSpecName "kube-api-access-rdvf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:47.326268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.325788 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvf2\" (UniqueName: \"kubernetes.io/projected/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-kube-api-access-rdvf2\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:47.326268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.325816 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d54c09-5e7d-4bff-a029-875ee3ab8d40-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:47.955058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.955026 4292 generic.go:356] "Generic (PLEG): container finished" podID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerID="5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8" exitCode=0 Oct 30 22:51:47.955265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.955070 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" Oct 30 22:51:47.955265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.955115 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" event={"ID":"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4","Type":"ContainerDied","Data":"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8"} Oct 30 22:51:47.955265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.955176 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd" event={"ID":"ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4","Type":"ContainerDied","Data":"2c1cba71c4052a1680dc4874f73db3695769c70c2eec86ee7b11b0c6e6cd6ba3"} Oct 30 22:51:47.955265 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.955207 4292 scope.go:122] "RemoveContainer" containerID="5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8" Oct 30 22:51:47.956553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.956427 4292 generic.go:356] "Generic (PLEG): container finished" podID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerID="2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34" exitCode=0 Oct 30 22:51:47.956553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.956492 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" Oct 30 22:51:47.956553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.956530 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" event={"ID":"e5d54c09-5e7d-4bff-a029-875ee3ab8d40","Type":"ContainerDied","Data":"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34"} Oct 30 22:51:47.956740 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.956601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26" event={"ID":"e5d54c09-5e7d-4bff-a029-875ee3ab8d40","Type":"ContainerDied","Data":"303eaa4740998d3aa677260a127c6d56ad8ffbb9fa76dbdac1a4ad22dfe25178"} Oct 30 22:51:47.965412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.965398 4292 scope.go:122] "RemoveContainer" containerID="5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8" Oct 30 22:51:47.965703 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:47.965678 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8\": container with ID starting with 5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8 not found: ID does not exist" containerID="5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8" Oct 30 22:51:47.965787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.965709 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8"} err="failed to get container status \"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8\": rpc error: code = NotFound desc = could not find container \"5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8\": container with ID starting with 5aeed2bffb3bc9594bf058bdf7310f8bd2407c13e3be9cf9d80cc6c7b83e3dc8 not found: ID does not exist" Oct 30 22:51:47.965787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.965728 4292 scope.go:122] "RemoveContainer" containerID="2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34" Oct 30 22:51:47.973970 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.973951 4292 scope.go:122] "RemoveContainer" containerID="2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34" Oct 30 22:51:47.974215 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:47.974197 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34\": container with ID starting with 2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34 not found: ID does not exist" containerID="2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34" Oct 30 22:51:47.974261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:47.974222 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34"} err="failed to get container status \"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34\": rpc error: code = NotFound desc = could not find container \"2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34\": container with ID starting with 2b075e1c0004bafdff76f5bf26061cd33e9f7df5244a8bfc9b7931fd31543b34 not found: ID does not exist" Oct 30 22:51:48.123815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:48.123742 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:51:48.174281 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:48.174251 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-72648e-5764b7cf95hctd"] Oct 30 22:51:48.241616 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:48.241588 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:51:48.247486 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:48.247464 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-72648e-7d44bxxm26"] Oct 30 22:51:50.145328 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:50.145295 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" path="/var/lib/kubelet/pods/e5d54c09-5e7d-4bff-a029-875ee3ab8d40/volumes" Oct 30 22:51:50.145723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:51:50.145682 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" path="/var/lib/kubelet/pods/ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4/volumes" Oct 30 22:51:56.640124 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:51:56.640082 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:52:05.056487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:52:05.056451 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x"] Oct 30 22:52:05.106756 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:52:05.106721 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2gr4x"] Oct 30 22:52:06.144357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:52:06.144322 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebcb4cb-5a73-46b7-ac70-de79402b6fef" path="/var/lib/kubelet/pods/1ebcb4cb-5a73-46b7-ac70-de79402b6fef/volumes" Oct 30 22:52:26.639352 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:52:26.639314 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:52:38.324452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:52:38.324420 4292 scope.go:122] "RemoveContainer" containerID="3347f60f8cb8a558d62675118027021b648300325bd1a6e6ca86b1e63936e153" Oct 30 22:52:38.332641 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:52:38.332624 4292 scope.go:122] "RemoveContainer" containerID="c9304617ebcc52eb67a5bc2e2bf7dd8a3e3f885bcbf3923ce48b6fe05121a2f0" Oct 30 22:52:56.638929 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:52:56.638889 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:53:26.684364 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:53:26.684325 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:53:28.039599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:28.035684 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:53:28.039599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:28.035766 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:53:30.145984 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.145954 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l"] Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146314 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146325 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146335 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" containerName="revoke-keys" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146340 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" containerName="revoke-keys" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146349 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146354 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146365 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" Oct 30 22:53:30.146386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146369 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146395 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146401 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146409 4292 cpu_manager.go:414] "containerMap: removing container" podUID="20955bf4-a275-402e-9311-69834ea66fc2" containerName="main" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146414 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20955bf4-a275-402e-9311-69834ea66fc2" containerName="main" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146478 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146484 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e8be32-00f7-484c-a6b5-c39f70cdfe6f" containerName="revoke-keys" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146493 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc7f7a8-5fd1-4690-a8f7-fdebd02af4a9" containerName="istio-proxy" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146500 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4bd886-94a8-4a0a-ac4e-b97f53eb01b4" containerName="payload-processing" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146507 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d54c09-5e7d-4bff-a029-875ee3ab8d40" containerName="payload-pre-processing" Oct 30 22:53:30.146738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.146512 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="20955bf4-a275-402e-9311-69834ea66fc2" containerName="main" Oct 30 22:53:30.149767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.149751 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.190672 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.190642 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjx2n\" (UniqueName: \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.190808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.190677 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.190808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.190707 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.202153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.202128 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l"] Oct 30 22:53:30.250368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.250345 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:53:30.250368 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.250364 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:53:30.250547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.250365 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:53:30.254408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.254385 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:53:30.291618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.291596 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjx2n\" (UniqueName: \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.291723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.291628 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.291723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.291663 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.292226 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.292209 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.293881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.293863 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.354741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.354721 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjx2n\" (UniqueName: \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\") pod \"maas-api-revoke-sub-d576e68b-pzj8l\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.462760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.462700 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:30.836458 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:30.836432 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l"] Oct 30 22:53:30.838372 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:53:30.838345 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704d7fc7_1d4c_4a4e_a3bf_0b7b82bb9090.slice/crio-dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59 WatchSource:0}: Error finding container dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59: Status 404 returned error can't find the container with id dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59 Oct 30 22:53:31.474675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:31.474554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" event={"ID":"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090","Type":"ContainerStarted","Data":"941f6b2782e5883ab4cd95530428db231d238a84b56948bb80475d4fb5e2f1e3"} Oct 30 22:53:31.474675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:31.474640 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" event={"ID":"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090","Type":"ContainerStarted","Data":"dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59"} Oct 30 22:53:31.596585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:31.596523 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" podStartSLOduration=1.596510243 podStartE2EDuration="1.596510243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:31.59478881 +0000 UTC m=+3539.563608374" watchObservedRunningTime="2026-10-30 22:53:31.596510243 +0000 UTC m=+3539.565329806" Oct 30 22:53:32.480541 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:32.480505 4292 generic.go:356] "Generic (PLEG): container finished" podID="704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" containerID="941f6b2782e5883ab4cd95530428db231d238a84b56948bb80475d4fb5e2f1e3" exitCode=0 Oct 30 22:53:32.481056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:32.480616 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" event={"ID":"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090","Type":"ContainerDied","Data":"941f6b2782e5883ab4cd95530428db231d238a84b56948bb80475d4fb5e2f1e3"} Oct 30 22:53:33.621153 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.621126 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:33.718657 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.718633 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\") pod \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " Oct 30 22:53:33.718812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.718668 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\" (UniqueName: \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\") pod \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " Oct 30 22:53:33.718812 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.718717 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\") pod \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\" (UID: \"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090\") " Oct 30 22:53:33.719062 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.719034 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca" pod "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" (UID: "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:33.719165 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.719105 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:33.720741 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.720718 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca" pod "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" (UID: "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:33.720851 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.720789 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n" pod "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" (UID: "704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090"). InnerVolumeSpecName "kube-api-access-bjx2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:33.819617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.819567 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjx2n\" (UniqueName: \"kubernetes.io/projected/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-kube-api-access-bjx2n\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:33.819745 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:33.819621 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:34.491434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:34.491345 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" event={"ID":"704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090","Type":"ContainerDied","Data":"dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59"} Oct 30 22:53:34.491434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:34.491371 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l" Oct 30 22:53:34.491434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:34.491394 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dda61ac503dfdacb02fddb2ddf1671c58bafef564d59c567214630463afbdb59" Oct 30 22:53:38.365723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:38.365692 4292 scope.go:122] "RemoveContainer" containerID="848fe17737400e6f2ec70aa358c7164f2609c072fe1bf859c1f86af217d620de" Oct 30 22:53:38.374285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:38.374258 4292 scope.go:122] "RemoveContainer" containerID="28b90a2628cbd465c5688d94b11e872e321b429f0cb647e0e295adc1644fa14b" Oct 30 22:53:39.056171 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:39.056140 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt"] Oct 30 22:53:39.106681 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:39.106649 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qnbgt"] Oct 30 22:53:40.145511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:40.145469 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1a5e61-81e9-4d76-961d-998923e3f06f" path="/var/lib/kubelet/pods/5d1a5e61-81e9-4d76-961d-998923e3f06f/volumes" Oct 30 22:53:49.074594 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:49.074498 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv"] Oct 30 22:53:49.132868 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:49.132832 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-92ctv"] Oct 30 22:53:50.158598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:50.157659 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1" path="/var/lib/kubelet/pods/e9e78d1c-be93-4b87-a2e1-a666a5c5c9d1/volumes" Oct 30 22:53:56.639392 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:53:56.639356 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:53:59.326935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.326897 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28"] Oct 30 22:53:59.327382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.327370 4292 cpu_manager.go:414] "containerMap: removing container" podUID="704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" containerName="revoke-keys" Oct 30 22:53:59.327431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.327386 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" containerName="revoke-keys" Oct 30 22:53:59.327465 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.327445 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" containerName="revoke-keys" Oct 30 22:53:59.330463 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.330447 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.434929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.434899 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:53:59.435256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.435217 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:53:59.435361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.435241 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:53:59.435361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.435259 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:53:59.437191 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.437172 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28"] Oct 30 22:53:59.448143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.448120 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.448258 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.448230 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4n9\" (UniqueName: \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.448312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.448271 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.549442 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.549412 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.549629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.549487 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4n9\" (UniqueName: \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.549629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.549526 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.550216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.550195 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.551733 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.551702 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.612366 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.612304 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4n9\" (UniqueName: \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\") pod \"maas-api-revoke-sub-d2bc93bc-nvd28\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:53:59.643245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:53:59.643218 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:54:00.020064 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:00.020038 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28"] Oct 30 22:54:00.022201 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:54:00.022174 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a61737d_6421_49f3_ac2b_ed70dd903fcf.slice/crio-42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91 WatchSource:0}: Error finding container 42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91: Status 404 returned error can't find the container with id 42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91 Oct 30 22:54:00.614125 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:00.614086 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" event={"ID":"5a61737d-6421-49f3-ac2b-ed70dd903fcf","Type":"ContainerStarted","Data":"ea90cd02cbd647839fe8a358e189486b21a5972348d75a6ebfc9ae5cf987236a"} Oct 30 22:54:00.614503 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:00.614148 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" event={"ID":"5a61737d-6421-49f3-ac2b-ed70dd903fcf","Type":"ContainerStarted","Data":"42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91"} Oct 30 22:54:00.734874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:00.734829 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" podStartSLOduration=1.7348172160000002 podStartE2EDuration="1.734817216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:00.732974261 +0000 UTC m=+3568.701793827" watchObservedRunningTime="2026-10-30 22:54:00.734817216 +0000 UTC m=+3568.703636813" Oct 30 22:54:01.620586 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:01.620549 4292 generic.go:356] "Generic (PLEG): container finished" podID="5a61737d-6421-49f3-ac2b-ed70dd903fcf" containerID="ea90cd02cbd647839fe8a358e189486b21a5972348d75a6ebfc9ae5cf987236a" exitCode=0 Oct 30 22:54:01.620992 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:01.620635 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" event={"ID":"5a61737d-6421-49f3-ac2b-ed70dd903fcf","Type":"ContainerDied","Data":"ea90cd02cbd647839fe8a358e189486b21a5972348d75a6ebfc9ae5cf987236a"} Oct 30 22:54:02.767525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.767500 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:54:02.879244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.879178 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\") pod \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " Oct 30 22:54:02.879244 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.879227 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\" (UniqueName: \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\") pod \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " Oct 30 22:54:02.879457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.879313 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\") pod \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\" (UID: \"5a61737d-6421-49f3-ac2b-ed70dd903fcf\") " Oct 30 22:54:02.879671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.879648 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca" pod "5a61737d-6421-49f3-ac2b-ed70dd903fcf" (UID: "5a61737d-6421-49f3-ac2b-ed70dd903fcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:02.881143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.881115 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca" pod "5a61737d-6421-49f3-ac2b-ed70dd903fcf" (UID: "5a61737d-6421-49f3-ac2b-ed70dd903fcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:02.881296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.881273 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9" pod "5a61737d-6421-49f3-ac2b-ed70dd903fcf" (UID: "5a61737d-6421-49f3-ac2b-ed70dd903fcf"). InnerVolumeSpecName "kube-api-access-nb4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:02.980201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.980177 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4n9\" (UniqueName: \"kubernetes.io/projected/5a61737d-6421-49f3-ac2b-ed70dd903fcf-kube-api-access-nb4n9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:02.980201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.980201 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:02.980358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:02.980213 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61737d-6421-49f3-ac2b-ed70dd903fcf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:03.630787 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:03.630755 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" Oct 30 22:54:03.631047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:03.630781 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28" event={"ID":"5a61737d-6421-49f3-ac2b-ed70dd903fcf","Type":"ContainerDied","Data":"42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91"} Oct 30 22:54:03.631047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:03.630829 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42471f4e49fddd40907dfa9c8168cf610cec4e6da650a2723eac44ac4587cf91" Oct 30 22:54:12.363483 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.363446 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:54:12.363897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.363886 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5a61737d-6421-49f3-ac2b-ed70dd903fcf" containerName="revoke-keys" Oct 30 22:54:12.363942 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.363899 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a61737d-6421-49f3-ac2b-ed70dd903fcf" containerName="revoke-keys" Oct 30 22:54:12.364009 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.363996 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a61737d-6421-49f3-ac2b-ed70dd903fcf" containerName="revoke-keys" Oct 30 22:54:12.367150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.367123 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.421958 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.421924 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c38ecf-openshift-default-dockercfg-fpvr6\"" Oct 30 22:54:12.457660 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457620 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.457822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457731 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.457822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457768 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvtk\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.457948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457903 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.458012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457964 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.458012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.457995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.458143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.458120 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.458183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.458168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.458262 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.458246 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.473882 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.473858 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:54:12.559320 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559281 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559327 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559350 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvtk\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559380 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559405 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559422 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559500 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559485 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559518 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559549 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.559846 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559769 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.560011 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.559966 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.560074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.560036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.560121 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.560097 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.560360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.560335 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.561917 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.561889 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.562083 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.562068 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.623154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.623082 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.623289 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.623163 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvtk\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\") pod \"e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:12.686374 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:12.686344 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:13.119309 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.119285 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:54:13.121120 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:54:13.121096 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8eb96c_470e_4fcd_ad50_db513145c660.slice/crio-05b0b6546ee485252d1566a77995a9b291088aa30f7c9bd31f918d0fdc2875ea WatchSource:0}: Error finding container 05b0b6546ee485252d1566a77995a9b291088aa30f7c9bd31f918d0fdc2875ea: Status 404 returned error can't find the container with id 05b0b6546ee485252d1566a77995a9b291088aa30f7c9bd31f918d0fdc2875ea Oct 30 22:54:13.123324 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.123291 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:13.123391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.123358 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:13.123391 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.123386 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:13.681987 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.681954 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerStarted","Data":"c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9"} Oct 30 22:54:13.682376 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.682009 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerStarted","Data":"05b0b6546ee485252d1566a77995a9b291088aa30f7c9bd31f918d0fdc2875ea"} Oct 30 22:54:13.687221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.687195 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:13.812863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:13.812809 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podStartSLOduration=1.812795606 podStartE2EDuration="1.812795606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:13.810937929 +0000 UTC m=+3581.779757494" watchObservedRunningTime="2026-10-30 22:54:13.812795606 +0000 UTC m=+3581.781615164" Oct 30 22:54:14.691675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:14.691640 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:15.692941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:15.692905 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:15.693842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:15.693822 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:20.591180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.591146 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:54:20.594693 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.594676 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.648995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.648968 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c38ecf\"" Oct 30 22:54:20.698727 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.698686 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9d655f11-dockercfg-lmwvk\"" Oct 30 22:54:20.701252 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.701225 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:54:20.739115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.739083 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.739312 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.739290 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2h5\" (UniqueName: \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.751013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.750987 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:54:20.754490 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.754472 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:20.789383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.789356 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:54:20.840772 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.840744 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:20.840899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.840789 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:20.840899 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.840887 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2h5\" (UniqueName: \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.840981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.840931 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.841603 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.841540 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.904512 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.904480 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2h5\" (UniqueName: \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\") pod \"payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.907245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.907220 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:20.941410 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.941380 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:20.941561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.941543 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:20.942429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:20.942404 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:21.005110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.005076 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\") pod \"payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:21.067675 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.067645 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:21.283386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.283362 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:54:21.284661 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:54:21.284632 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abce16c_c3c6_4e18_a74a_b453da79b662.slice/crio-9f728dc36f61fa0aef083eab93a957027de4ddb0f6e1e7e92698e30b8d0f7334 WatchSource:0}: Error finding container 9f728dc36f61fa0aef083eab93a957027de4ddb0f6e1e7e92698e30b8d0f7334: Status 404 returned error can't find the container with id 9f728dc36f61fa0aef083eab93a957027de4ddb0f6e1e7e92698e30b8d0f7334 Oct 30 22:54:21.440890 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.440862 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:54:21.443504 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:54:21.443474 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0445c487_6713_4715_8f38_85520a85f3b2.slice/crio-729171e4669206aada55eb56a44e37649fbfe86e14cdd6a13ceb87abb7801834 WatchSource:0}: Error finding container 729171e4669206aada55eb56a44e37649fbfe86e14cdd6a13ceb87abb7801834: Status 404 returned error can't find the container with id 729171e4669206aada55eb56a44e37649fbfe86e14cdd6a13ceb87abb7801834 Oct 30 22:54:21.724996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.724891 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" event={"ID":"0abce16c-c3c6-4e18-a74a-b453da79b662","Type":"ContainerStarted","Data":"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434"} Oct 30 22:54:21.724996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.724968 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:21.724996 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.724991 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" event={"ID":"0abce16c-c3c6-4e18-a74a-b453da79b662","Type":"ContainerStarted","Data":"9f728dc36f61fa0aef083eab93a957027de4ddb0f6e1e7e92698e30b8d0f7334"} Oct 30 22:54:21.726447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.726423 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" event={"ID":"0445c487-6713-4715-8f38-85520a85f3b2","Type":"ContainerStarted","Data":"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9"} Oct 30 22:54:21.726614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.726466 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" event={"ID":"0445c487-6713-4715-8f38-85520a85f3b2","Type":"ContainerStarted","Data":"729171e4669206aada55eb56a44e37649fbfe86e14cdd6a13ceb87abb7801834"} Oct 30 22:54:21.726614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.726542 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:21.845700 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.845656 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" podStartSLOduration=1.84564431 podStartE2EDuration="1.84564431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:21.844144427 +0000 UTC m=+3589.812963992" watchObservedRunningTime="2026-10-30 22:54:21.84564431 +0000 UTC m=+3589.814463874" Oct 30 22:54:21.969803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:21.969738 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" podStartSLOduration=1.969720069 podStartE2EDuration="1.969720069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:21.966649866 +0000 UTC m=+3589.935469427" watchObservedRunningTime="2026-10-30 22:54:21.969720069 +0000 UTC m=+3589.938539633" Oct 30 22:54:26.638975 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:54:26.638940 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:54:32.734476 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:32.734445 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:54:32.734910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:32.734682 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:54:38.409632 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:38.406812 4292 scope.go:122] "RemoveContainer" containerID="942f4e9080a024004317202fb42fbe585ba8d1534938d94bda7ac66fc47c3a85" Oct 30 22:54:38.418322 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:38.418293 4292 scope.go:122] "RemoveContainer" containerID="b245bc0c62ae823f166cdfe665534e3ac3f0983aa75f85b9971a210d17e809f0" Oct 30 22:54:44.979151 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:44.979115 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:54:44.984030 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:44.984011 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.050611 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.050550 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:54:45.069713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.069676 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.069879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.069725 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.069938 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.069900 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.069995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.069962 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngl8\" (UniqueName: \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.070071 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.070048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.070115 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.070084 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.091259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.091233 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c38ecf\"/\"kube-root-ca.crt\"" Oct 30 22:54:45.091259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.091234 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c38ecf\"/\"openshift-service-ca.crt\"" Oct 30 22:54:45.091415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.091280 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c38ecf\"/\"default-dockercfg-7hxbj\"" Oct 30 22:54:45.091415 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.091357 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c38ecf\"/\"simulator-single-main-c38ecf-kserve-self-signed-certs\"" Oct 30 22:54:45.171385 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171347 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171408 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngl8\" (UniqueName: \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171444 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171470 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171509 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171539 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.171940 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.171849 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.172041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.172022 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.172093 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.172020 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.173724 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.173697 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.174078 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.174061 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.234879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.234815 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngl8\" (UniqueName: \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\") pod \"simulator-single-main-c38ecf-kserve-574dcfcc94-57bct\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.295506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.295483 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:45.677025 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.676992 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:54:45.680205 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:54:45.680172 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b0b676_592a_4fbc_9f74_b6a0a5e0bce8.slice/crio-5bf31b4ee8bd2fc608731b9ff3381366862f6a88b0f3dbba595e10188b7dd453 WatchSource:0}: Error finding container 5bf31b4ee8bd2fc608731b9ff3381366862f6a88b0f3dbba595e10188b7dd453: Status 404 returned error can't find the container with id 5bf31b4ee8bd2fc608731b9ff3381366862f6a88b0f3dbba595e10188b7dd453 Oct 30 22:54:45.864926 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.864887 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" event={"ID":"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8","Type":"ContainerStarted","Data":"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059"} Oct 30 22:54:45.865096 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.864933 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" event={"ID":"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8","Type":"ContainerStarted","Data":"5bf31b4ee8bd2fc608731b9ff3381366862f6a88b0f3dbba595e10188b7dd453"} Oct 30 22:54:45.988285 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:45.988239 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" podStartSLOduration=1.988226957 podStartE2EDuration="1.988226957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:45.98671403 +0000 UTC m=+3613.955533597" watchObservedRunningTime="2026-10-30 22:54:45.988226957 +0000 UTC m=+3613.957046521" Oct 30 22:54:46.066519 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.066487 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": read tcp 10.134.0.2:45778->10.134.1.12:15021: read: connection reset by peer" start-of-body= Oct 30 22:54:46.066673 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.066550 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": read tcp 10.134.0.2:45778->10.134.1.12:15021: read: connection reset by peer" Oct 30 22:54:46.296327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.296295 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:46.871419 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.871382 4292 generic.go:356] "Generic (PLEG): container finished" podID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerID="c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9" exitCode=0 Oct 30 22:54:46.871622 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.871454 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerDied","Data":"c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9"} Oct 30 22:54:46.871914 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.871900 4292 scope.go:122] "RemoveContainer" containerID="c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9" Oct 30 22:54:46.872941 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.872917 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:46.873004 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.872993 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:46.873041 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:46.873027 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:47.878771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:47.878734 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerStarted","Data":"32df88a963c0d7b975174fe591ce123f541ee382a17df761514ac4b8e069086b"} Oct 30 22:54:48.690947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:48.690911 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:48.691618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:48.691297 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:54:48.691618 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:48.691348 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 22:54:49.689349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:49.688914 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:54:49.689349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:49.688975 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 22:54:50.703960 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:50.703920 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:54:50.704542 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:50.704507 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 22:54:51.696024 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:51.695982 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:54:51.696342 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:51.696311 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 22:54:52.697599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:52.695881 4292 patch_prober.go:29] interesting pod/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:54:52.697599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:52.695948 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 22:54:53.690084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:53.690054 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:53.690351 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:53.690326 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:53.690928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:53.690912 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:54:55.295838 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:55.295804 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:55.308261 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:55.308237 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:55.955207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:54:55.955181 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:54:56.638380 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:54:56.638347 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:55:26.639492 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:55:26.639446 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:55:31.013850 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.013817 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm"] Oct 30 22:55:31.017682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.017664 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.019056 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.019030 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.019150 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.019068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.019221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.019200 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9zf\" (UniqueName: \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.076797 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.076772 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm"] Oct 30 22:55:31.119910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.119881 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.119910 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.119912 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.120098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.119958 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9zf\" (UniqueName: \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.124777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.124755 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:55:31.125109 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.125092 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:55:31.125154 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.125141 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:55:31.125203 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.125191 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:55:31.131563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.131532 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.133286 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.133269 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.183403 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.183383 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9zf\" (UniqueName: \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\") pod \"maas-api-revoke-sub-7389aaae-rsjfm\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.330084 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.330058 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:31.708474 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:31.708450 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm"] Oct 30 22:55:31.710166 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:55:31.710134 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2602e803_1eff_4651_9425_3d8e778883eb.slice/crio-472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b WatchSource:0}: Error finding container 472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b: Status 404 returned error can't find the container with id 472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b Oct 30 22:55:32.120967 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:32.120938 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" event={"ID":"2602e803-1eff-4651-9425-3d8e778883eb","Type":"ContainerStarted","Data":"04eccb31b7abb5ba1ffa34b27711aa58aa638ec3961aaf0a42ca30aedf7b10ac"} Oct 30 22:55:32.121335 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:32.120985 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" event={"ID":"2602e803-1eff-4651-9425-3d8e778883eb","Type":"ContainerStarted","Data":"472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b"} Oct 30 22:55:32.241492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:32.241444 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" podStartSLOduration=2.241429728 podStartE2EDuration="2.241429728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:32.239924645 +0000 UTC m=+3660.208744208" watchObservedRunningTime="2026-10-30 22:55:32.241429728 +0000 UTC m=+3660.210249292" Oct 30 22:55:33.127692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:33.127660 4292 generic.go:356] "Generic (PLEG): container finished" podID="2602e803-1eff-4651-9425-3d8e778883eb" containerID="04eccb31b7abb5ba1ffa34b27711aa58aa638ec3961aaf0a42ca30aedf7b10ac" exitCode=0 Oct 30 22:55:33.128081 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:33.127743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" event={"ID":"2602e803-1eff-4651-9425-3d8e778883eb","Type":"ContainerDied","Data":"04eccb31b7abb5ba1ffa34b27711aa58aa638ec3961aaf0a42ca30aedf7b10ac"} Oct 30 22:55:34.258095 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.258069 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:34.349293 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.349269 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\" (UniqueName: \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\") pod \"2602e803-1eff-4651-9425-3d8e778883eb\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " Oct 30 22:55:34.349428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.349310 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\") pod \"2602e803-1eff-4651-9425-3d8e778883eb\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " Oct 30 22:55:34.349428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.349364 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\") pod \"2602e803-1eff-4651-9425-3d8e778883eb\" (UID: \"2602e803-1eff-4651-9425-3d8e778883eb\") " Oct 30 22:55:34.349734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.349708 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca" pod "2602e803-1eff-4651-9425-3d8e778883eb" (UID: "2602e803-1eff-4651-9425-3d8e778883eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:34.351497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.351472 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca" pod "2602e803-1eff-4651-9425-3d8e778883eb" (UID: "2602e803-1eff-4651-9425-3d8e778883eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:34.351637 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.351615 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf" pod "2602e803-1eff-4651-9425-3d8e778883eb" (UID: "2602e803-1eff-4651-9425-3d8e778883eb"). InnerVolumeSpecName "kube-api-access-dd9zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:34.451314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.451241 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.451314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.451279 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9zf\" (UniqueName: \"kubernetes.io/projected/2602e803-1eff-4651-9425-3d8e778883eb-kube-api-access-dd9zf\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.451314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:34.451294 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2602e803-1eff-4651-9425-3d8e778883eb-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:35.139332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:35.139294 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" event={"ID":"2602e803-1eff-4651-9425-3d8e778883eb","Type":"ContainerDied","Data":"472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b"} Oct 30 22:55:35.139332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:35.139319 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm" Oct 30 22:55:35.139332 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:35.139340 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="472496c279b45bff4841657eff1fbf86ce768ba8775b80c96dc1b22860d77a6b" Oct 30 22:55:38.442382 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:38.442344 4292 scope.go:122] "RemoveContainer" containerID="ed594165bb796eeae83efa35971217297d2ddef0eaf689500799667f76a84d8d" Oct 30 22:55:38.451216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:38.451188 4292 scope.go:122] "RemoveContainer" containerID="12035c416c67df9e40a299ef436fa92b593b4317868541d379f51ae76b5d45e0" Oct 30 22:55:42.752773 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.752739 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj"] Oct 30 22:55:42.753216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.753202 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2602e803-1eff-4651-9425-3d8e778883eb" containerName="revoke-keys" Oct 30 22:55:42.753277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.753217 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2602e803-1eff-4651-9425-3d8e778883eb" containerName="revoke-keys" Oct 30 22:55:42.753315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.753293 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2602e803-1eff-4651-9425-3d8e778883eb" containerName="revoke-keys" Oct 30 22:55:42.756383 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.756367 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.824412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.824376 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.824626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.824471 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.824626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.824501 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj"] Oct 30 22:55:42.824626 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.824514 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddjt\" (UniqueName: \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.872341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.872305 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:55:42.872341 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.872327 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:55:42.872539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.872388 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:55:42.872539 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.872456 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:55:42.925211 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.925180 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.925373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.925330 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.925448 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.925394 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddjt\" (UniqueName: \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.925952 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.925925 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.927511 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.927488 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:42.988734 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:42.988710 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddjt\" (UniqueName: \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:43.069540 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:43.069519 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:43.444338 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:43.444315 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj"] Oct 30 22:55:43.447137 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:55:43.447107 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbef4e5f_f43b_45dc_a99f_83f73a515c80.slice/crio-fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009 WatchSource:0}: Error finding container fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009: Status 404 returned error can't find the container with id fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009 Oct 30 22:55:44.189735 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:44.189702 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" event={"ID":"fbef4e5f-f43b-45dc-a99f-83f73a515c80","Type":"ContainerStarted","Data":"2c9ab059f7832e9f469d83d3d4f3c303d5e0b2c41aaa02fae6ff239332243511"} Oct 30 22:55:44.190155 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:44.189752 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" event={"ID":"fbef4e5f-f43b-45dc-a99f-83f73a515c80","Type":"ContainerStarted","Data":"fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009"} Oct 30 22:55:44.309975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:44.309926 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" podStartSLOduration=2.309912635 podStartE2EDuration="2.309912635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:44.309369285 +0000 UTC m=+3672.278188850" watchObservedRunningTime="2026-10-30 22:55:44.309912635 +0000 UTC m=+3672.278732582" Oct 30 22:55:45.195974 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:45.195941 4292 generic.go:356] "Generic (PLEG): container finished" podID="fbef4e5f-f43b-45dc-a99f-83f73a515c80" containerID="2c9ab059f7832e9f469d83d3d4f3c303d5e0b2c41aaa02fae6ff239332243511" exitCode=0 Oct 30 22:55:45.196380 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:45.196013 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" event={"ID":"fbef4e5f-f43b-45dc-a99f-83f73a515c80","Type":"ContainerDied","Data":"2c9ab059f7832e9f469d83d3d4f3c303d5e0b2c41aaa02fae6ff239332243511"} Oct 30 22:55:46.330127 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.330100 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:46.459671 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.459610 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\") pod \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " Oct 30 22:55:46.459803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.459674 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\") pod \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " Oct 30 22:55:46.459803 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.459738 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\" (UniqueName: \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\") pod \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\" (UID: \"fbef4e5f-f43b-45dc-a99f-83f73a515c80\") " Oct 30 22:55:46.460013 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.459993 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca" pod "fbef4e5f-f43b-45dc-a99f-83f73a515c80" (UID: "fbef4e5f-f43b-45dc-a99f-83f73a515c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:46.460123 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.460106 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:46.461888 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.461860 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt" pod "fbef4e5f-f43b-45dc-a99f-83f73a515c80" (UID: "fbef4e5f-f43b-45dc-a99f-83f73a515c80"). InnerVolumeSpecName "kube-api-access-pddjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:46.461999 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.461872 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca" pod "fbef4e5f-f43b-45dc-a99f-83f73a515c80" (UID: "fbef4e5f-f43b-45dc-a99f-83f73a515c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:46.560894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.560865 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbef4e5f-f43b-45dc-a99f-83f73a515c80-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:46.560894 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:46.560891 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pddjt\" (UniqueName: \"kubernetes.io/projected/fbef4e5f-f43b-45dc-a99f-83f73a515c80-kube-api-access-pddjt\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.206751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:47.206717 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" event={"ID":"fbef4e5f-f43b-45dc-a99f-83f73a515c80","Type":"ContainerDied","Data":"fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009"} Oct 30 22:55:47.206912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:47.206765 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1dccdadf92c1c4bad670e95cd720e111b27e96ee18606d4b3c6d11a2569009" Oct 30 22:55:47.206912 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:47.206733 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj" Oct 30 22:55:47.221303 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:47.221279 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:55:54.272144 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:54.272112 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:55:54.379777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:54.379735 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:55:56.638686 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:55:56.638649 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:55:59.272562 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:59.272518 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" containerID="cri-o://8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9" gracePeriod=25 Oct 30 22:55:59.380780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:55:59.380743 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" containerID="cri-o://8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434" gracePeriod=25 Oct 30 22:56:01.055716 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:01.055680 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr"] Oct 30 22:56:01.105880 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:01.105848 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dabdfd-dvkhr"] Oct 30 22:56:02.145214 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.145177 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b7bf78-5d77-4293-a203-be21692a3ba2" path="/var/lib/kubelet/pods/06b7bf78-5d77-4293-a203-be21692a3ba2/volumes" Oct 30 22:56:02.240927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.240885 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" podUID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" containerName="main" containerID="cri-o://accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059" gracePeriod=45 Oct 30 22:56:02.495530 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.495507 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:56:02.610847 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.610805 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.610955 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\" (UniqueName: \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.610980 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\" (UniqueName: \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611021 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611011 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611046 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611048 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:02.611166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611084 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\") pod \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\" (UID: \"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8\") " Oct 30 22:56:02.611326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611304 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:02.611420 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611393 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:02.611554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611491 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.611554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611510 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.611554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.611522 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.613255 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.613232 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:02.613339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.613288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:02.613339 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.613317 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8" pod "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" (UID: "21b0b676-592a-4fbc-9f74-b6a0a5e0bce8"). InnerVolumeSpecName "kube-api-access-bngl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:02.713077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.713007 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngl8\" (UniqueName: \"kubernetes.io/projected/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-kube-api-access-bngl8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.713077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.713036 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.713077 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.713046 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:02.733682 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.733645 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:02.733843 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:02.733820 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:03.286240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.286206 4292 generic.go:356] "Generic (PLEG): container finished" podID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" containerID="accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059" exitCode=0 Oct 30 22:56:03.286240 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.286244 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" event={"ID":"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8","Type":"ContainerDied","Data":"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059"} Oct 30 22:56:03.286816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.286269 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" event={"ID":"21b0b676-592a-4fbc-9f74-b6a0a5e0bce8","Type":"ContainerDied","Data":"5bf31b4ee8bd2fc608731b9ff3381366862f6a88b0f3dbba595e10188b7dd453"} Oct 30 22:56:03.286816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.286245 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct" Oct 30 22:56:03.286816 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.286285 4292 scope.go:122] "RemoveContainer" containerID="accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059" Oct 30 22:56:03.295553 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.295533 4292 scope.go:122] "RemoveContainer" containerID="accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059" Oct 30 22:56:03.295949 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:03.295920 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059\": container with ID starting with accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059 not found: ID does not exist" containerID="accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059" Oct 30 22:56:03.296075 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.295959 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059"} err="failed to get container status \"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059\": rpc error: code = NotFound desc = could not find container \"accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059\": container with ID starting with accaa46dffec6062f0cd69dd5e59bd0a8df253ecb66dcd9a07a49e002803e059 not found: ID does not exist" Oct 30 22:56:03.456517 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.456486 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:56:03.506869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:03.506840 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c38ecf/simulator-single-main-c38ecf-kserve-574dcfcc94-57bct"] Oct 30 22:56:04.150848 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:04.150805 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" path="/var/lib/kubelet/pods/21b0b676-592a-4fbc-9f74-b6a0a5e0bce8/volumes" Oct 30 22:56:05.238604 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:05.238558 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj"] Oct 30 22:56:05.289112 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:05.289080 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c38ecf-xwjlj"] Oct 30 22:56:06.145513 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:06.145479 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbef4e5f-f43b-45dc-a99f-83f73a515c80" path="/var/lib/kubelet/pods/fbef4e5f-f43b-45dc-a99f-83f73a515c80/volumes" Oct 30 22:56:12.732881 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:12.732847 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:12.733276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:12.732847 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:13.117808 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:13.117766 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:56:13.118029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:13.118000 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" containerID="cri-o://32df88a963c0d7b975174fe591ce123f541ee382a17df761514ac4b8e069086b" gracePeriod=30 Oct 30 22:56:18.360068 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.360039 4292 generic.go:356] "Generic (PLEG): container finished" podID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerID="32df88a963c0d7b975174fe591ce123f541ee382a17df761514ac4b8e069086b" exitCode=0 Oct 30 22:56:18.360429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.360113 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerDied","Data":"32df88a963c0d7b975174fe591ce123f541ee382a17df761514ac4b8e069086b"} Oct 30 22:56:18.360429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.360176 4292 scope.go:122] "RemoveContainer" containerID="c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9" Oct 30 22:56:18.380178 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:18.380149 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9\": container with ID starting with c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9 not found: ID does not exist" containerID="c32e4b94cc3b85f96d5122067e7ea7513254e6553a3c22d299ad4fb060f788b9" Oct 30 22:56:18.383674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.383617 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:56:18.477593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477548 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477624 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477678 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477725 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477741 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.477778 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477759 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.478034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477847 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:18.478034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477878 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.478034 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.477935 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\") pod \"8a8eb96c-470e-4fcd-ad50-db513145c660\" (UID: \"8a8eb96c-470e-4fcd-ad50-db513145c660\") " Oct 30 22:56:18.478172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478131 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:18.478172 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478153 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:18.478306 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478269 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:18.478436 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478410 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:18.478534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478437 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.478534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478473 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8eb96c-470e-4fcd-ad50-db513145c660-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.478534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478485 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.478534 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.478496 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.480066 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.480044 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:18.480194 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.480109 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:18.480444 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.480421 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:56:18.480509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.480465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk" pod "8a8eb96c-470e-4fcd-ad50-db513145c660" (UID: "8a8eb96c-470e-4fcd-ad50-db513145c660"). InnerVolumeSpecName "kube-api-access-jqvtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:18.579599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.579559 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.579599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.579598 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8eb96c-470e-4fcd-ad50-db513145c660-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.579804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.579608 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.579804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.579617 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvtk\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-kube-api-access-jqvtk\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:18.579804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:18.579625 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8eb96c-470e-4fcd-ad50-db513145c660-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:19.365804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.365765 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" event={"ID":"8a8eb96c-470e-4fcd-ad50-db513145c660","Type":"ContainerDied","Data":"05b0b6546ee485252d1566a77995a9b291088aa30f7c9bd31f918d0fdc2875ea"} Oct 30 22:56:19.366200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.365836 4292 scope.go:122] "RemoveContainer" containerID="32df88a963c0d7b975174fe591ce123f541ee382a17df761514ac4b8e069086b" Oct 30 22:56:19.366200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.365779 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt" Oct 30 22:56:19.540241 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.540170 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:56:19.586885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.586855 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:56:19.590994 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.590964 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\") pod \"0445c487-6713-4715-8f38-85520a85f3b2\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " Oct 30 22:56:19.591164 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.591042 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\") pod \"0445c487-6713-4715-8f38-85520a85f3b2\" (UID: \"0445c487-6713-4715-8f38-85520a85f3b2\") " Oct 30 22:56:19.591497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.591468 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume" pod "0445c487-6713-4715-8f38-85520a85f3b2" (UID: "0445c487-6713-4715-8f38-85520a85f3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:19.595117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.595087 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj" pod "0445c487-6713-4715-8f38-85520a85f3b2" (UID: "0445c487-6713-4715-8f38-85520a85f3b2"). InnerVolumeSpecName "kube-api-access-slqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:19.614179 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.614160 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:56:19.637381 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.637354 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c38ecf-openshift-default-55b5bf8cc-rbgpt"] Oct 30 22:56:19.692162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692142 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\") pod \"0abce16c-c3c6-4e18-a74a-b453da79b662\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " Oct 30 22:56:19.692259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692183 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\" (UniqueName: \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\") pod \"0abce16c-c3c6-4e18-a74a-b453da79b662\" (UID: \"0abce16c-c3c6-4e18-a74a-b453da79b662\") " Oct 30 22:56:19.692471 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692449 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume" pod "0abce16c-c3c6-4e18-a74a-b453da79b662" (UID: "0abce16c-c3c6-4e18-a74a-b453da79b662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:19.692554 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692540 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/0445c487-6713-4715-8f38-85520a85f3b2-kube-api-access-slqxj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:19.692613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692557 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0445c487-6713-4715-8f38-85520a85f3b2-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:19.692613 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.692567 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abce16c-c3c6-4e18-a74a-b453da79b662-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:19.694162 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.694143 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5" pod "0abce16c-c3c6-4e18-a74a-b453da79b662" (UID: "0abce16c-c3c6-4e18-a74a-b453da79b662"). InnerVolumeSpecName "kube-api-access-6n2h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:19.793373 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:19.793293 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2h5\" (UniqueName: \"kubernetes.io/projected/0abce16c-c3c6-4e18-a74a-b453da79b662-kube-api-access-6n2h5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.043501 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043425 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:56:20.043889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043874 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043890 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043913 4292 cpu_manager.go:414] "containerMap: removing container" podUID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" containerName="main" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043918 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" containerName="main" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043930 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043935 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.043943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043942 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fbef4e5f-f43b-45dc-a99f-83f73a515c80" containerName="revoke-keys" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043947 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbef4e5f-f43b-45dc-a99f-83f73a515c80" containerName="revoke-keys" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043958 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043962 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043972 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.043977 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044031 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044038 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbef4e5f-f43b-45dc-a99f-83f73a515c80" containerName="revoke-keys" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044047 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerName="payload-pre-processing" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044054 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" containerName="istio-proxy" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044061 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0445c487-6713-4715-8f38-85520a85f3b2" containerName="payload-processing" Oct 30 22:56:20.044137 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.044070 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b0b676-592a-4fbc-9f74-b6a0a5e0bce8" containerName="main" Oct 30 22:56:20.047120 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.047102 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096117 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096089 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096121 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096307 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096193 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096324 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096404 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096368 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.096487 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.096471 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zkb\" (UniqueName: \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.101597 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.101555 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4ab1c912-kserve-self-signed-certs\"" Oct 30 22:56:20.107898 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.107874 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:56:20.146251 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.146221 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8eb96c-470e-4fcd-ad50-db513145c660" path="/var/lib/kubelet/pods/8a8eb96c-470e-4fcd-ad50-db513145c660/volumes" Oct 30 22:56:20.197216 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197224 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197274 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197316 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197412 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197378 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zkb\" (UniqueName: \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197777 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197752 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197828 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197809 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.197879 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.197850 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.199617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.199593 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.199815 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.199800 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.260548 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.260526 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zkb\" (UniqueName: \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\") pod \"e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.358438 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.358409 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:20.374016 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.373987 4292 generic.go:356] "Generic (PLEG): container finished" podID="0abce16c-c3c6-4e18-a74a-b453da79b662" containerID="8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434" exitCode=0 Oct 30 22:56:20.374386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.374049 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" Oct 30 22:56:20.374386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.374076 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" event={"ID":"0abce16c-c3c6-4e18-a74a-b453da79b662","Type":"ContainerDied","Data":"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434"} Oct 30 22:56:20.374386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.374131 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc" event={"ID":"0abce16c-c3c6-4e18-a74a-b453da79b662","Type":"ContainerDied","Data":"9f728dc36f61fa0aef083eab93a957027de4ddb0f6e1e7e92698e30b8d0f7334"} Oct 30 22:56:20.374386 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.374160 4292 scope.go:122] "RemoveContainer" containerID="8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434" Oct 30 22:56:20.382889 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.382858 4292 generic.go:356] "Generic (PLEG): container finished" podID="0445c487-6713-4715-8f38-85520a85f3b2" containerID="8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9" exitCode=0 Oct 30 22:56:20.383225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.382972 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" event={"ID":"0445c487-6713-4715-8f38-85520a85f3b2","Type":"ContainerDied","Data":"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9"} Oct 30 22:56:20.383225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.383025 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" event={"ID":"0445c487-6713-4715-8f38-85520a85f3b2","Type":"ContainerDied","Data":"729171e4669206aada55eb56a44e37649fbfe86e14cdd6a13ceb87abb7801834"} Oct 30 22:56:20.383225 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.383104 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg" Oct 30 22:56:20.392723 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.392701 4292 scope.go:122] "RemoveContainer" containerID="8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434" Oct 30 22:56:20.393029 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:20.393006 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434\": container with ID starting with 8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434 not found: ID does not exist" containerID="8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434" Oct 30 22:56:20.393098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.393038 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434"} err="failed to get container status \"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434\": rpc error: code = NotFound desc = could not find container \"8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434\": container with ID starting with 8952a67e0bbf369558a1039e3588cf0444d67b1c053ce451f7a5ad654d767434 not found: ID does not exist" Oct 30 22:56:20.393098 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.393056 4292 scope.go:122] "RemoveContainer" containerID="8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9" Oct 30 22:56:20.402166 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.402147 4292 scope.go:122] "RemoveContainer" containerID="8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9" Oct 30 22:56:20.402498 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:20.402428 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9\": container with ID starting with 8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9 not found: ID does not exist" containerID="8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9" Oct 30 22:56:20.402498 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.402461 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9"} err="failed to get container status \"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9\": rpc error: code = NotFound desc = could not find container \"8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9\": container with ID starting with 8ac1d5f25e85e43778392b1b75bb50fc5870cf9f4b6c31a237e19b52c5ae8dd9 not found: ID does not exist" Oct 30 22:56:20.542823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.542789 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:56:20.548434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.548406 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c38ecf-fb8bbh6trc"] Oct 30 22:56:20.661497 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.661435 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:56:20.668852 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.668824 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c38ecf-5d6df749d6m5sg"] Oct 30 22:56:20.741945 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:20.741920 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:56:20.743496 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:56:20.743463 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f92a44_beb4_4b79_9dd4_8f0ff41896ec.slice/crio-50e8836270ffb3b8eda78f17ea7ceb087ff0cb4a6cb14811ab62c23eda349ce6 WatchSource:0}: Error finding container 50e8836270ffb3b8eda78f17ea7ceb087ff0cb4a6cb14811ab62c23eda349ce6: Status 404 returned error can't find the container with id 50e8836270ffb3b8eda78f17ea7ceb087ff0cb4a6cb14811ab62c23eda349ce6 Oct 30 22:56:21.392327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:21.392291 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" event={"ID":"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec","Type":"ContainerStarted","Data":"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c"} Oct 30 22:56:21.392327 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:21.392331 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" event={"ID":"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec","Type":"ContainerStarted","Data":"50e8836270ffb3b8eda78f17ea7ceb087ff0cb4a6cb14811ab62c23eda349ce6"} Oct 30 22:56:21.514061 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:21.514010 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" podStartSLOduration=2.513997938 podStartE2EDuration="2.513997938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:21.512988322 +0000 UTC m=+3709.481807886" watchObservedRunningTime="2026-10-30 22:56:21.513997938 +0000 UTC m=+3709.482817502" Oct 30 22:56:22.145235 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:22.145201 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0445c487-6713-4715-8f38-85520a85f3b2" path="/var/lib/kubelet/pods/0445c487-6713-4715-8f38-85520a85f3b2/volumes" Oct 30 22:56:22.145585 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:22.145559 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0abce16c-c3c6-4e18-a74a-b453da79b662" path="/var/lib/kubelet/pods/0abce16c-c3c6-4e18-a74a-b453da79b662/volumes" Oct 30 22:56:22.358947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:22.358895 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:26.755553 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:26.755512 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:56:28.046598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:28.043673 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:56:28.046598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:28.043755 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:56:30.359105 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:30.359065 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:30.372080 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:30.372055 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:30.461929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:56:30.461905 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:56:56.638630 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:56:56.638537 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:57:26.639869 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:57:26.639830 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:57:37.333738 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.333702 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8"] Oct 30 22:57:37.337360 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.337342 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.396760 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.396728 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8"] Oct 30 22:57:37.445200 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.445177 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:57:37.445357 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.445231 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:57:37.449598 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.449553 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.449705 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.449685 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdd9\" (UniqueName: \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.449771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.449757 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.486236 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.486215 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:57:37.495823 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.495805 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:57:37.550399 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.550364 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.550521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.550424 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdd9\" (UniqueName: \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.550521 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.550472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.551047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.551021 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.552883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.552861 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.614157 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.614103 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdd9\" (UniqueName: \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\") pod \"maas-api-revoke-sub-452b6d7b-krlw8\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:37.649918 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:37.649894 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:38.024640 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.024615 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8"] Oct 30 22:57:38.026510 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:57:38.026481 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0bbc9e_6d87_4dbf_bbd4_8e3119be38d6.slice/crio-9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf WatchSource:0}: Error finding container 9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf: Status 404 returned error can't find the container with id 9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf Oct 30 22:57:38.497326 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.497297 4292 scope.go:122] "RemoveContainer" containerID="a7e25a867ef8144ad2b8ca53730ad0205be058414f821d26c422d27254f3c565" Oct 30 22:57:38.505991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.505963 4292 scope.go:122] "RemoveContainer" containerID="036f6164b6664cb24078199a153ae251c3f0c7e39c4951cdb5c33aa1c151fa09" Oct 30 22:57:38.805593 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.805545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" event={"ID":"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6","Type":"ContainerStarted","Data":"9d41b8260c80d57ffcfa508a20392968282cfb115167c71f9f62ef8a832e2d12"} Oct 30 22:57:38.805798 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.805619 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" event={"ID":"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6","Type":"ContainerStarted","Data":"9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf"} Oct 30 22:57:38.928841 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:38.928795 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" podStartSLOduration=1.92878163 podStartE2EDuration="1.92878163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:38.927489942 +0000 UTC m=+3786.896309508" watchObservedRunningTime="2026-10-30 22:57:38.92878163 +0000 UTC m=+3786.897601194" Oct 30 22:57:39.814188 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:39.814151 4292 generic.go:356] "Generic (PLEG): container finished" podID="3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" containerID="9d41b8260c80d57ffcfa508a20392968282cfb115167c71f9f62ef8a832e2d12" exitCode=0 Oct 30 22:57:39.814717 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:39.814240 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" event={"ID":"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6","Type":"ContainerDied","Data":"9d41b8260c80d57ffcfa508a20392968282cfb115167c71f9f62ef8a832e2d12"} Oct 30 22:57:41.203885 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.203859 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:41.284102 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.284078 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\") pod \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " Oct 30 22:57:41.284245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.284144 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\") pod \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " Oct 30 22:57:41.284245 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.284213 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\" (UniqueName: \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\") pod \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\" (UID: \"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6\") " Oct 30 22:57:41.284408 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.284387 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca" pod "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" (UID: "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:41.320826 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.320782 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9" pod "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" (UID: "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6"). InnerVolumeSpecName "kube-api-access-5xdd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:41.335743 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.335690 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca" pod "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" (UID: "3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:41.391239 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.391204 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xdd9\" (UniqueName: \"kubernetes.io/projected/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-kube-api-access-5xdd9\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.391473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.391451 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.391629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.391612 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.861520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.861403 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" event={"ID":"3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6","Type":"ContainerDied","Data":"9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf"} Oct 30 22:57:41.861520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.861467 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e9e0355b38e439758a6ce1f5da960b7796c5c7ca585c2824a80f0e770e219cf" Oct 30 22:57:41.861520 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:41.861423 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8" Oct 30 22:57:50.169314 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:57:50.169281 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:57:56.675599 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:57:56.675549 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:58:05.188620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.188554 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" podUID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" containerName="main" containerID="cri-o://1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c" gracePeriod=45 Oct 30 22:58:05.437113 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.437087 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:58:05.543473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543403 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543473 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543472 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543500 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543531 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\" (UniqueName: \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543570 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543615 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\") pod \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\" (UID: \"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec\") " Oct 30 22:58:05.543786 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543724 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:05.544058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543847 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:05.544058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.543909 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:05.544058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.544021 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-model-cache\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:05.544058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.544038 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tmp-dir\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:05.544058 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.544052 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-home\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:05.545805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.545783 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:58:05.545805 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.545793 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "kube-api-access-f6zkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:05.545928 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.545788 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm" pod "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" (UID: "d6f92a44-beb4-4b79-9dd4-8f0ff41896ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:05.645183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.645158 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-dshm\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:05.645183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.645181 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-tls-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:05.645349 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:05.645190 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zkb\" (UniqueName: \"kubernetes.io/projected/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec-kube-api-access-f6zkb\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:06.003087 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.003051 4292 generic.go:356] "Generic (PLEG): container finished" podID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" containerID="1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c" exitCode=0 Oct 30 22:58:06.003301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.003097 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" Oct 30 22:58:06.003301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.003139 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" event={"ID":"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec","Type":"ContainerDied","Data":"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c"} Oct 30 22:58:06.003301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.003196 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6" event={"ID":"d6f92a44-beb4-4b79-9dd4-8f0ff41896ec","Type":"ContainerDied","Data":"50e8836270ffb3b8eda78f17ea7ceb087ff0cb4a6cb14811ab62c23eda349ce6"} Oct 30 22:58:06.003301 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.003219 4292 scope.go:122] "RemoveContainer" containerID="1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c" Oct 30 22:58:06.012101 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.012081 4292 scope.go:122] "RemoveContainer" containerID="1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c" Oct 30 22:58:06.012352 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:58:06.012334 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c\": container with ID starting with 1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c not found: ID does not exist" containerID="1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c" Oct 30 22:58:06.012409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.012362 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c"} err="failed to get container status \"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c\": rpc error: code = NotFound desc = could not find container \"1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c\": container with ID starting with 1204fec68f700bb007e0f98d9bd3fc1887f11848bb029de63e31060d57684e8c not found: ID does not exist" Oct 30 22:58:06.172450 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.172423 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:58:06.222872 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:06.222844 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4ab1c912-kserve-5855447c89-sczn6"] Oct 30 22:58:08.144692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:08.144655 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" path="/var/lib/kubelet/pods/d6f92a44-beb4-4b79-9dd4-8f0ff41896ec/volumes" Oct 30 22:58:11.471428 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471396 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p"] Oct 30 22:58:11.471883 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471868 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" containerName="main" Oct 30 22:58:11.471935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471884 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" containerName="main" Oct 30 22:58:11.471935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471910 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" containerName="revoke-keys" Oct 30 22:58:11.471935 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471915 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" containerName="revoke-keys" Oct 30 22:58:11.472047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471976 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" containerName="revoke-keys" Oct 30 22:58:11.472047 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.471990 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f92a44-beb4-4b79-9dd4-8f0ff41896ec" containerName="main" Oct 30 22:58:11.474991 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.474968 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.529953 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.529924 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:58:11.532208 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.532183 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:58:11.580751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.580721 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:58:11.580751 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.580736 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:58:11.582842 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.582821 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p"] Oct 30 22:58:11.595299 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.595275 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhljs\" (UniqueName: \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.595369 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.595312 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.595434 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.595414 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.696122 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.696099 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhljs\" (UniqueName: \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.696317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.696134 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.696317 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.696177 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.696775 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.696756 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.698447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.698428 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.759506 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.759452 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhljs\" (UniqueName: \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\") pod \"maas-api-revoke-sub-3e31a506-htf2p\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:11.788270 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:11.788239 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:12.161921 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:12.161892 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p"] Oct 30 22:58:12.163238 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:58:12.163200 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6fcfe5_f6ff_4740_b7c7_2d3e696c9acf.slice/crio-7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b WatchSource:0}: Error finding container 7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b: Status 404 returned error can't find the container with id 7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b Oct 30 22:58:13.040933 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:13.040891 4292 generic.go:356] "Generic (PLEG): container finished" podID="9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" containerID="9bfba70de9975e8af7cda31975af4d13b1e8b063a0f6c75a95a621f7ce1d3950" exitCode=0 Oct 30 22:58:13.041400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:13.041004 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" event={"ID":"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf","Type":"ContainerDied","Data":"9bfba70de9975e8af7cda31975af4d13b1e8b063a0f6c75a95a621f7ce1d3950"} Oct 30 22:58:13.041400 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:13.041052 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" event={"ID":"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf","Type":"ContainerStarted","Data":"7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b"} Oct 30 22:58:14.392822 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.392490 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:14.562799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.561885 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\" (UniqueName: \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\") pod \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " Oct 30 22:58:14.562799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.562082 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\") pod \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " Oct 30 22:58:14.562799 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.562117 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\") pod \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\" (UID: \"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf\") " Oct 30 22:58:14.563806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.563780 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca" pod "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" (UID: "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:58:14.566246 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.566219 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca" pod "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" (UID: "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:58:14.566493 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.566470 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs" pod "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" (UID: "9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf"). InnerVolumeSpecName "kube-api-access-qhljs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:14.664039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.663961 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:14.664039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.663988 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:14.664039 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:14.663997 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhljs\" (UniqueName: \"kubernetes.io/projected/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf-kube-api-access-qhljs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:15.059180 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:15.059153 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" Oct 30 22:58:15.059344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:15.059151 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p" event={"ID":"9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf","Type":"ContainerDied","Data":"7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b"} Oct 30 22:58:15.059344 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:15.059275 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f67cc9fbb3f2993241d9c3e627b4279aede6fc341e31f749f0992886cda726b" Oct 30 22:58:26.638083 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:58:26.638041 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:58:34.060108 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:34.060071 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l"] Oct 30 22:58:34.111729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:34.111699 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-pzj8l"] Oct 30 22:58:34.147197 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:34.147165 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090" path="/var/lib/kubelet/pods/704d7fc7-1d4c-4a4e-a3bf-0b7b82bb9090/volumes" Oct 30 22:58:44.294565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.294524 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr"] Oct 30 22:58:44.295029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.294999 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" containerName="revoke-keys" Oct 30 22:58:44.295029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.295012 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" containerName="revoke-keys" Oct 30 22:58:44.295110 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.295100 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" containerName="revoke-keys" Oct 30 22:58:44.298070 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.298054 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.352780 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.352756 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:58:44.366457 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.366432 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.366596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.366477 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46xv\" (UniqueName: \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.366596 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.366560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.402525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.402496 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:58:44.402525 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.402506 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:58:44.402771 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.402548 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4mjj\"" Oct 30 22:58:44.404742 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.404710 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr"] Oct 30 22:58:44.467263 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.467233 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.467409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.467281 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46xv\" (UniqueName: \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.467409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.467316 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.468036 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.468013 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.469655 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.469632 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.530649 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.530621 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46xv\" (UniqueName: \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\") pod \"maas-api-revoke-sub-7d62742f-44mqr\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.609943 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.609887 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:44.982472 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:44.982449 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr"] Oct 30 22:58:44.984137 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:58:44.984109 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bceec1_5457_4266_8e40_b561063c0afa.slice/crio-a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9 WatchSource:0}: Error finding container a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9: Status 404 returned error can't find the container with id a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9 Oct 30 22:58:45.247674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:45.247608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" event={"ID":"c1bceec1-5457-4266-8e40-b561063c0afa","Type":"ContainerStarted","Data":"2e47100455fbd87ea61c60aeec60420cee426085dc446277aa6edfa7b3c7075e"} Oct 30 22:58:45.247674 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:45.247668 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" event={"ID":"c1bceec1-5457-4266-8e40-b561063c0afa","Type":"ContainerStarted","Data":"a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9"} Oct 30 22:58:45.368348 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:45.368301 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" podStartSLOduration=1.368289651 podStartE2EDuration="1.368289651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:45.367103265 +0000 UTC m=+3853.335922831" watchObservedRunningTime="2026-10-30 22:58:45.368289651 +0000 UTC m=+3853.337109214" Oct 30 22:58:46.253414 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:46.253374 4292 generic.go:356] "Generic (PLEG): container finished" podID="c1bceec1-5457-4266-8e40-b561063c0afa" containerID="2e47100455fbd87ea61c60aeec60420cee426085dc446277aa6edfa7b3c7075e" exitCode=0 Oct 30 22:58:46.253634 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:46.253462 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" event={"ID":"c1bceec1-5457-4266-8e40-b561063c0afa","Type":"ContainerDied","Data":"2e47100455fbd87ea61c60aeec60420cee426085dc446277aa6edfa7b3c7075e"} Oct 30 22:58:47.395715 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.395690 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:47.489484 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.489453 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\") pod \"c1bceec1-5457-4266-8e40-b561063c0afa\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " Oct 30 22:58:47.489667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.489525 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\") pod \"c1bceec1-5457-4266-8e40-b561063c0afa\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " Oct 30 22:58:47.489667 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.489619 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\" (UniqueName: \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\") pod \"c1bceec1-5457-4266-8e40-b561063c0afa\" (UID: \"c1bceec1-5457-4266-8e40-b561063c0afa\") " Oct 30 22:58:47.489927 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.489905 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca" pod "c1bceec1-5457-4266-8e40-b561063c0afa" (UID: "c1bceec1-5457-4266-8e40-b561063c0afa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:58:47.491690 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.491664 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca" pod "c1bceec1-5457-4266-8e40-b561063c0afa" (UID: "c1bceec1-5457-4266-8e40-b561063c0afa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:58:47.491863 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.491700 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv" pod "c1bceec1-5457-4266-8e40-b561063c0afa" (UID: "c1bceec1-5457-4266-8e40-b561063c0afa"). InnerVolumeSpecName "kube-api-access-j46xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:47.590985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.590950 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:47.590985 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.590987 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46xv\" (UniqueName: \"kubernetes.io/projected/c1bceec1-5457-4266-8e40-b561063c0afa-kube-api-access-j46xv\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:47.591184 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:47.591002 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bceec1-5457-4266-8e40-b561063c0afa-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:58:48.264712 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:48.264678 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" Oct 30 22:58:48.264975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:48.264713 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr" event={"ID":"c1bceec1-5457-4266-8e40-b561063c0afa","Type":"ContainerDied","Data":"a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9"} Oct 30 22:58:48.264975 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:58:48.264771 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e284e24ddcc53b7e6bf6ecba70c1331bce3123083ca06167baf41196bd59c9" Oct 30 22:58:56.639657 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:58:56.639617 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:59:03.058407 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:03.058372 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28"] Oct 30 22:59:03.108470 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:03.108439 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nvd28"] Oct 30 22:59:04.145063 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:04.145024 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a61737d-6421-49f3-ac2b-ed70dd903fcf" path="/var/lib/kubelet/pods/5a61737d-6421-49f3-ac2b-ed70dd903fcf/volumes" Oct 30 22:59:05.045555 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.045515 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 22:59:05.046221 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.046201 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c1bceec1-5457-4266-8e40-b561063c0afa" containerName="revoke-keys" Oct 30 22:59:05.046280 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.046225 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bceec1-5457-4266-8e40-b561063c0afa" containerName="revoke-keys" Oct 30 22:59:05.046356 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.046344 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bceec1-5457-4266-8e40-b561063c0afa" containerName="revoke-keys" Oct 30 22:59:05.050431 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.050405 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.105620 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.105568 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b8bfacd-openshift-default-dockercfg-97hms\"" Oct 30 22:59:05.140714 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.140674 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.140869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.140723 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.140869 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.140831 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.140951 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.140870 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.140995 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.140967 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.141059 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.141036 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.141277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.141254 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.141392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.141323 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.141392 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.141359 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.156976 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.156954 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 22:59:05.241978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.241945 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.241985 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242034 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242055 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242077 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242183 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242107 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242256 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242349 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242401 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242489 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242404 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242709 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242490 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242774 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242752 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242827 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242772 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.242929 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.242912 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.244375 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.244358 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.244630 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.244613 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.312452 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.312392 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.312559 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.312545 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\") pod \"e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.366447 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.366418 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:05.792449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.792421 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 22:59:05.793293 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:59:05.793263 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fe6107_3b32_448f_a00e_408b6304d99c.slice/crio-5b50dac94b6dc97bc1a86d6e852b421ab53f6fa1d7c0b63c45e28b1072076e55 WatchSource:0}: Error finding container 5b50dac94b6dc97bc1a86d6e852b421ab53f6fa1d7c0b63c45e28b1072076e55: Status 404 returned error can't find the container with id 5b50dac94b6dc97bc1a86d6e852b421ab53f6fa1d7c0b63c45e28b1072076e55 Oct 30 22:59:05.795469 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.795433 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:05.795547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.795495 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:05.795547 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:05.795541 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:06.356959 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:06.356914 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerStarted","Data":"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b"} Oct 30 22:59:06.357430 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:06.357010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerStarted","Data":"5b50dac94b6dc97bc1a86d6e852b421ab53f6fa1d7c0b63c45e28b1072076e55"} Oct 30 22:59:06.367276 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:06.367252 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:06.372177 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:06.372153 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:06.483429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:06.483382 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podStartSLOduration=2.483369724 podStartE2EDuration="2.483369724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:06.482789346 +0000 UTC m=+3874.451608911" watchObservedRunningTime="2026-10-30 22:59:06.483369724 +0000 UTC m=+3874.452189288" Oct 30 22:59:07.362029 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:07.361999 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:07.362789 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:07.362769 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:12.591981 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.591938 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 22:59:12.595874 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.595851 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.654561 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.654539 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 22:59:12.702139 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.702116 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b8bfacd\"" Oct 30 22:59:12.702256 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.702148 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b8bfacd-dockercfg-pm6p2\"" Oct 30 22:59:12.709855 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.709826 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.709939 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.709895 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s5q\" (UniqueName: \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.754509 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.754476 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 22:59:12.758143 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.758122 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.811361 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.811325 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.811492 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.811456 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqhd\" (UniqueName: \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.811563 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.811543 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s5q\" (UniqueName: \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.811713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.811694 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.812565 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.812536 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.813728 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.813709 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 22:59:12.874989 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.874929 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s5q\" (UniqueName: \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\") pod \"payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.909215 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.909192 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:12.912633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.912282 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.912633 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.912344 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqhd\" (UniqueName: \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.913001 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.912983 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:12.976713 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:12.976663 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqhd\" (UniqueName: \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\") pod \"payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:13.071617 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.071568 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:13.284131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.284102 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 22:59:13.285334 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:59:13.285308 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b115414_ceca_4273_9a16_a00d535b2af6.slice/crio-10adee869564479c1bca14cce7a081466d977f04320afdfc70267945f458179a WatchSource:0}: Error finding container 10adee869564479c1bca14cce7a081466d977f04320afdfc70267945f458179a: Status 404 returned error can't find the container with id 10adee869564479c1bca14cce7a081466d977f04320afdfc70267945f458179a Oct 30 22:59:13.391277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.391188 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" event={"ID":"6b115414-ceca-4273-9a16-a00d535b2af6","Type":"ContainerStarted","Data":"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a"} Oct 30 22:59:13.391277 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.391259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" event={"ID":"6b115414-ceca-4273-9a16-a00d535b2af6","Type":"ContainerStarted","Data":"10adee869564479c1bca14cce7a081466d977f04320afdfc70267945f458179a"} Oct 30 22:59:13.392358 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.392334 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:13.451388 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.451368 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 22:59:13.453636 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:59:13.453612 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527f5aa_2d8f_4633_b7e2_416f7eec1a2f.slice/crio-33b6ba01213c70822cac3319999bfc0823e9542fe0ad548d58697da1f3a50a02 WatchSource:0}: Error finding container 33b6ba01213c70822cac3319999bfc0823e9542fe0ad548d58697da1f3a50a02: Status 404 returned error can't find the container with id 33b6ba01213c70822cac3319999bfc0823e9542fe0ad548d58697da1f3a50a02 Oct 30 22:59:13.512702 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:13.512660 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" podStartSLOduration=1.512646883 podStartE2EDuration="1.512646883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:13.511930324 +0000 UTC m=+3881.480749888" watchObservedRunningTime="2026-10-30 22:59:13.512646883 +0000 UTC m=+3881.481466446" Oct 30 22:59:14.397268 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:14.397227 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" event={"ID":"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f","Type":"ContainerStarted","Data":"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea"} Oct 30 22:59:14.397686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:14.397291 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" event={"ID":"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f","Type":"ContainerStarted","Data":"33b6ba01213c70822cac3319999bfc0823e9542fe0ad548d58697da1f3a50a02"} Oct 30 22:59:14.397686 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:14.397409 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:14.519687 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:14.519638 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" podStartSLOduration=2.519625627 podStartE2EDuration="2.519625627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:14.517766913 +0000 UTC m=+3882.486586477" watchObservedRunningTime="2026-10-30 22:59:14.519625627 +0000 UTC m=+3882.488445190" Oct 30 22:59:25.410363 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:25.410329 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 22:59:25.410862 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:25.410416 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 22:59:26.639635 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:59:26.639597 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:59:31.480806 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.480768 4292 generic.go:356] "Generic (PLEG): container finished" podID="28fe6107-3b32-448f-a00e-408b6304d99c" containerID="c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b" exitCode=0 Oct 30 22:59:31.481296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.480838 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerDied","Data":"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b"} Oct 30 22:59:31.481296 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.481175 4292 scope.go:122] "RemoveContainer" containerID="c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b" Oct 30 22:59:31.482222 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.482195 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:31.482271 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.482264 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:31.482315 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:31.482295 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:59:32.492319 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:32.492272 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerStarted","Data":"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170"} Oct 30 22:59:33.366947 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:33.366909 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:33.368118 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:33.368087 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:59:33.368266 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:33.368145 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 22:59:34.367449 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:34.367409 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:59:34.367978 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:34.367471 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 22:59:35.367683 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:35.367648 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:59:35.368074 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:35.367713 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 22:59:36.366905 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:36.366869 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:59:36.367106 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:36.366943 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 22:59:37.362079 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:37.362044 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:37.367692 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:37.367670 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:59:37.367779 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:37.367712 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 22:59:38.371543 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:38.371513 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:38.372409 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:38.372389 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 22:59:38.539955 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:38.539929 4292 scope.go:122] "RemoveContainer" containerID="941f6b2782e5883ab4cd95530428db231d238a84b56948bb80475d4fb5e2f1e3" Oct 30 22:59:45.281645 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.281555 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw"] Oct 30 22:59:45.285210 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.285191 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.323897 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.323874 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.324012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.323923 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.324012 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.324001 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ctg\" (UniqueName: \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.344767 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.344743 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw"] Oct 30 22:59:45.392082 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.392057 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:59:45.392207 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.392187 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:59:45.392259 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.392231 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:59:45.392297 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.392289 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 22:59:45.425429 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.425409 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.425526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.425437 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ctg\" (UniqueName: \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.425526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.425513 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.426073 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.426045 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.427804 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.427784 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.489054 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.489026 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ctg\" (UniqueName: \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\") pod \"maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.597670 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.597648 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:45.968097 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:45.968070 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw"] Oct 30 22:59:45.970249 ip-10-0-142-210 kubenswrapper[4292]: W1030 22:59:45.970225 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aa2616_1d7c_45a9_8d32_1f23590a6bfa.slice/crio-6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17 WatchSource:0}: Error finding container 6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17: Status 404 returned error can't find the container with id 6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17 Oct 30 22:59:46.574201 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:46.574163 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" event={"ID":"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa","Type":"ContainerStarted","Data":"7bfc5f30f7c77bd673b0734bfb8f4af2b7480468b5d720a653d0d89b48aea9f3"} Oct 30 22:59:46.574599 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:46.574222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" event={"ID":"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa","Type":"ContainerStarted","Data":"6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17"} Oct 30 22:59:46.693931 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:46.693885 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" podStartSLOduration=1.693872159 podStartE2EDuration="1.693872159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:46.692933856 +0000 UTC m=+3914.661753420" watchObservedRunningTime="2026-10-30 22:59:46.693872159 +0000 UTC m=+3914.662691723" Oct 30 22:59:47.581243 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:47.581202 4292 generic.go:356] "Generic (PLEG): container finished" podID="b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" containerID="7bfc5f30f7c77bd673b0734bfb8f4af2b7480468b5d720a653d0d89b48aea9f3" exitCode=0 Oct 30 22:59:47.581739 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:47.581285 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" event={"ID":"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa","Type":"ContainerDied","Data":"7bfc5f30f7c77bd673b0734bfb8f4af2b7480468b5d720a653d0d89b48aea9f3"} Oct 30 22:59:48.719858 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.719834 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:48.756494 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.756467 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\") pod \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " Oct 30 22:59:48.756614 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.756594 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\") pod \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " Oct 30 22:59:48.756729 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.756713 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\" (UniqueName: \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\") pod \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\" (UID: \"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa\") " Oct 30 22:59:48.756871 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.756852 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca" pod "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" (UID: "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:48.757051 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.757032 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:48.758526 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.758498 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca" pod "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" (UID: "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:59:48.758677 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.758653 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg" pod "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" (UID: "b8aa2616-1d7c-45a9-8d32-1f23590a6bfa"). InnerVolumeSpecName "kube-api-access-t4ctg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:48.857948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.857879 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:48.857948 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:48.857905 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ctg\" (UniqueName: \"kubernetes.io/projected/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa-kube-api-access-t4ctg\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:49.592131 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:49.592098 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" Oct 30 22:59:49.592304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:49.592132 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw" event={"ID":"b8aa2616-1d7c-45a9-8d32-1f23590a6bfa","Type":"ContainerDied","Data":"6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17"} Oct 30 22:59:49.592304 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:49.592181 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6760c0e292c6f19c7129ce8c5a4e2346dcfb7c8b1072f4127531f81715ffee17" Oct 30 22:59:56.639652 ip-10-0-142-210 kubenswrapper[4292]: E1030 22:59:56.639611 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 22:59:56.771629 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:56.771594 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 22:59:56.881829 ip-10-0-142-210 kubenswrapper[4292]: I1030 22:59:56.881801 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 23:00:00.168180 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.168139 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:00.168636 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.168554 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" containerName="revoke-keys" Oct 30 23:00:00.168636 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.168566 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" containerName="revoke-keys" Oct 30 23:00:00.168723 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.168664 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" containerName="revoke-keys" Oct 30 23:00:00.171950 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.171930 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.173145 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.173123 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:00.176773 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.176757 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.231395 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.231366 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:00.252120 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.252091 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.252278 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.252135 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrgs\" (UniqueName: \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.252278 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.252245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h25fj\" (UniqueName: \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.252385 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.252287 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.278635 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.278594 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 23:00:00.278790 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.278627 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:00:00.278790 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.278627 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:00:00.281360 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.281333 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:00.353020 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.352985 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.353230 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.353037 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrgs\" (UniqueName: \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.353230 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.353095 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h25fj\" (UniqueName: \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.353230 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.353135 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.353695 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.353670 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.353877 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.353856 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.420685 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.420601 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h25fj\" (UniqueName: \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\") pod \"maas-api-key-cleanup-29890020-wtvp9\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.423136 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.423112 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrgs\" (UniqueName: \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\") pod \"maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.486058 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.486026 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:00.492262 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.492241 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:00.879136 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.879104 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:00.880236 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:00:00.880210 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c453db_8174_4ad3_ae01_c568c6ff6e21.slice/crio-d4abefdf958e10f57406aa07c7ccf001d22617b54a2fa93a97dd20e3ea688f4e WatchSource:0}: Error finding container d4abefdf958e10f57406aa07c7ccf001d22617b54a2fa93a97dd20e3ea688f4e: Status 404 returned error can't find the container with id d4abefdf958e10f57406aa07c7ccf001d22617b54a2fa93a97dd20e3ea688f4e Oct 30 23:00:00.925130 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:00.925104 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:00.925733 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:00:00.925706 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5be3f0_bee8_4e4f_a64f_21aa4a38b6ba.slice/crio-481c5ed05c412c44d025cfe9c384ad78cadd67707d7c547eb2f1e0a1ec16aa1b WatchSource:0}: Error finding container 481c5ed05c412c44d025cfe9c384ad78cadd67707d7c547eb2f1e0a1ec16aa1b: Status 404 returned error can't find the container with id 481c5ed05c412c44d025cfe9c384ad78cadd67707d7c547eb2f1e0a1ec16aa1b Oct 30 23:00:01.655194 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.655164 4292 generic.go:356] "Generic (PLEG): container finished" podID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerID="0922e6a23f74986276148a0de6020facf80ac735f6f78a3d3e42f7cf36a59758" exitCode=60 Oct 30 23:00:01.655559 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.655253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" event={"ID":"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba","Type":"ContainerDied","Data":"0922e6a23f74986276148a0de6020facf80ac735f6f78a3d3e42f7cf36a59758"} Oct 30 23:00:01.655559 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.655313 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" event={"ID":"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba","Type":"ContainerStarted","Data":"481c5ed05c412c44d025cfe9c384ad78cadd67707d7c547eb2f1e0a1ec16aa1b"} Oct 30 23:00:01.655559 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.655496 4292 scope.go:122] "RemoveContainer" containerID="0922e6a23f74986276148a0de6020facf80ac735f6f78a3d3e42f7cf36a59758" Oct 30 23:00:01.656917 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.656886 4292 generic.go:356] "Generic (PLEG): container finished" podID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerID="0f29ebb0771ff0e832a85059b1929df78d30f5b54132206863930217d83cd51d" exitCode=60 Oct 30 23:00:01.656987 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.656965 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerDied","Data":"0f29ebb0771ff0e832a85059b1929df78d30f5b54132206863930217d83cd51d"} Oct 30 23:00:01.657028 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.657012 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerStarted","Data":"d4abefdf958e10f57406aa07c7ccf001d22617b54a2fa93a97dd20e3ea688f4e"} Oct 30 23:00:01.657240 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.657227 4292 scope.go:122] "RemoveContainer" containerID="0f29ebb0771ff0e832a85059b1929df78d30f5b54132206863930217d83cd51d" Oct 30 23:00:01.772506 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.772460 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" podUID="6b115414-ceca-4273-9a16-a00d535b2af6" containerName="payload-processing" containerID="cri-o://3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a" gracePeriod=25 Oct 30 23:00:01.883132 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:01.883023 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" podUID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" containerName="payload-pre-processing" containerID="cri-o://93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea" gracePeriod=25 Oct 30 23:00:02.013886 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.013853 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 23:00:02.075924 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.075890 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\") pod \"6b115414-ceca-4273-9a16-a00d535b2af6\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " Oct 30 23:00:02.076119 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.075958 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\" (UniqueName: \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\") pod \"6b115414-ceca-4273-9a16-a00d535b2af6\" (UID: \"6b115414-ceca-4273-9a16-a00d535b2af6\") " Oct 30 23:00:02.076258 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.076233 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume" pod "6b115414-ceca-4273-9a16-a00d535b2af6" (UID: "6b115414-ceca-4273-9a16-a00d535b2af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:02.078052 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.078032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q" pod "6b115414-ceca-4273-9a16-a00d535b2af6" (UID: "6b115414-ceca-4273-9a16-a00d535b2af6"). InnerVolumeSpecName "kube-api-access-r8s5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:02.143428 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.143358 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 23:00:02.178022 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.177980 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b115414-ceca-4273-9a16-a00d535b2af6-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:02.178022 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.178020 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8s5q\" (UniqueName: \"kubernetes.io/projected/6b115414-ceca-4273-9a16-a00d535b2af6-kube-api-access-r8s5q\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:02.278926 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.278894 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\") pod \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " Oct 30 23:00:02.279112 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.279051 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\" (UniqueName: \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\") pod \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\" (UID: \"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f\") " Oct 30 23:00:02.279311 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.279289 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume" pod "3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" (UID: "3527f5aa-2d8f-4633-b7e2-416f7eec1a2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:02.281238 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.281210 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd" pod "3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" (UID: "3527f5aa-2d8f-4633-b7e2-416f7eec1a2f"). InnerVolumeSpecName "kube-api-access-2wqhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:02.380807 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.380764 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqhd\" (UniqueName: \"kubernetes.io/projected/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-kube-api-access-2wqhd\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:02.380807 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.380800 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:02.664609 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.664542 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" event={"ID":"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba","Type":"ContainerStarted","Data":"def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db"} Oct 30 23:00:02.665792 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.665765 4292 generic.go:356] "Generic (PLEG): container finished" podID="6b115414-ceca-4273-9a16-a00d535b2af6" containerID="3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a" exitCode=0 Oct 30 23:00:02.665888 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.665798 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" Oct 30 23:00:02.665888 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.665843 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" event={"ID":"6b115414-ceca-4273-9a16-a00d535b2af6","Type":"ContainerDied","Data":"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a"} Oct 30 23:00:02.666027 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.665897 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs" event={"ID":"6b115414-ceca-4273-9a16-a00d535b2af6","Type":"ContainerDied","Data":"10adee869564479c1bca14cce7a081466d977f04320afdfc70267945f458179a"} Oct 30 23:00:02.666027 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.665929 4292 scope.go:122] "RemoveContainer" containerID="3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a" Oct 30 23:00:02.667886 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.667706 4292 generic.go:356] "Generic (PLEG): container finished" podID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerID="e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c" exitCode=60 Oct 30 23:00:02.667886 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.667794 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerDied","Data":"e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c"} Oct 30 23:00:02.668066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.667986 4292 scope.go:122] "RemoveContainer" containerID="e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c" Oct 30 23:00:02.668218 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:02.668196 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-wtvp9_odh-ai-gateway-infra(86c453db-8174-4ad3-ae01-c568c6ff6e21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" Oct 30 23:00:02.677177 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.677102 4292 generic.go:356] "Generic (PLEG): container finished" podID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" containerID="93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea" exitCode=0 Oct 30 23:00:02.677177 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.677167 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" Oct 30 23:00:02.677353 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.677241 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" event={"ID":"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f","Type":"ContainerDied","Data":"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea"} Oct 30 23:00:02.677353 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.677320 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl" event={"ID":"3527f5aa-2d8f-4633-b7e2-416f7eec1a2f","Type":"ContainerDied","Data":"33b6ba01213c70822cac3319999bfc0823e9542fe0ad548d58697da1f3a50a02"} Oct 30 23:00:02.684657 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.684630 4292 scope.go:122] "RemoveContainer" containerID="3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a" Oct 30 23:00:02.685227 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:02.684987 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a\": container with ID starting with 3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a not found: ID does not exist" containerID="3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a" Oct 30 23:00:02.685227 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.685020 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a"} err="failed to get container status \"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a\": rpc error: code = NotFound desc = could not find container \"3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a\": container with ID starting with 3479b739cfff0d89310e69e1582176b1f71eec94cc5e8275947df1d9e78d040a not found: ID does not exist" Oct 30 23:00:02.685227 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.685046 4292 scope.go:122] "RemoveContainer" containerID="0f29ebb0771ff0e832a85059b1929df78d30f5b54132206863930217d83cd51d" Oct 30 23:00:02.751511 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.751490 4292 scope.go:122] "RemoveContainer" containerID="93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea" Oct 30 23:00:02.759792 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.759771 4292 scope.go:122] "RemoveContainer" containerID="93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea" Oct 30 23:00:02.760071 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:02.760050 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea\": container with ID starting with 93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea not found: ID does not exist" containerID="93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea" Oct 30 23:00:02.760117 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.760083 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea"} err="failed to get container status \"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea\": rpc error: code = NotFound desc = could not find container \"93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea\": container with ID starting with 93f6189f56e50638029fd39be97f4e273a8c747fb614462ede062792477c39ea not found: ID does not exist" Oct 30 23:00:02.788149 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.788102 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" podStartSLOduration=2.788087428 podStartE2EDuration="2.788087428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:02.786560155 +0000 UTC m=+3930.755379719" watchObservedRunningTime="2026-10-30 23:00:02.788087428 +0000 UTC m=+3930.756906992" Oct 30 23:00:02.902597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.902545 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 23:00:02.905991 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:02.905965 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b8bfacd-8bbb86845-mlshs"] Oct 30 23:00:03.018762 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.018685 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 23:00:03.024743 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.024710 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b8bfacd-69bdb46859-c4drl"] Oct 30 23:00:03.683735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.683702 4292 generic.go:356] "Generic (PLEG): container finished" podID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerID="def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db" exitCode=60 Oct 30 23:00:03.684222 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.683790 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" event={"ID":"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba","Type":"ContainerDied","Data":"def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db"} Oct 30 23:00:03.684222 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.683856 4292 scope.go:122] "RemoveContainer" containerID="0922e6a23f74986276148a0de6020facf80ac735f6f78a3d3e42f7cf36a59758" Oct 30 23:00:03.684222 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.684018 4292 scope.go:122] "RemoveContainer" containerID="def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db" Oct 30 23:00:03.684385 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:03.684294 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6_odh-ai-gateway-infra(7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" Oct 30 23:00:03.687007 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.686989 4292 scope.go:122] "RemoveContainer" containerID="e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c" Oct 30 23:00:03.687277 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:03.687254 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-wtvp9_odh-ai-gateway-infra(86c453db-8174-4ad3-ae01-c568c6ff6e21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" Oct 30 23:00:03.731206 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:03.731186 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:04.146068 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:04.146034 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" path="/var/lib/kubelet/pods/3527f5aa-2d8f-4633-b7e2-416f7eec1a2f/volumes" Oct 30 23:00:04.146424 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:04.146410 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b115414-ceca-4273-9a16-a00d535b2af6" path="/var/lib/kubelet/pods/6b115414-ceca-4273-9a16-a00d535b2af6/volumes" Oct 30 23:00:04.694348 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:04.694320 4292 scope.go:122] "RemoveContainer" containerID="def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db" Oct 30 23:00:04.694742 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:04.694603 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6_odh-ai-gateway-infra(7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" Oct 30 23:00:05.727983 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:05.727952 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw"] Oct 30 23:00:05.777448 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:05.777408 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b8bfacd-cc8nw"] Oct 30 23:00:05.844673 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:05.844652 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:06.017158 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.017085 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\") pod \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " Oct 30 23:00:06.017158 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.017122 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\" (UniqueName: \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\") pod \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\" (UID: \"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba\") " Oct 30 23:00:06.017422 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.017401 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca" pod "7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" (UID: "7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:06.019192 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.019164 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs" pod "7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" (UID: "7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba"). InnerVolumeSpecName "kube-api-access-cbrgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:06.117902 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.117875 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:06.117902 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.117899 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrgs\" (UniqueName: \"kubernetes.io/projected/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba-kube-api-access-cbrgs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:06.145570 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.145544 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8aa2616-1d7c-45a9-8d32-1f23590a6bfa" path="/var/lib/kubelet/pods/b8aa2616-1d7c-45a9-8d32-1f23590a6bfa/volumes" Oct 30 23:00:06.457662 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.457629 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 23:00:06.457845 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.457827 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" containerID="cri-o://d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170" gracePeriod=30 Oct 30 23:00:06.704721 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.704683 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" Oct 30 23:00:06.704721 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.704701 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6" event={"ID":"7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba","Type":"ContainerDied","Data":"481c5ed05c412c44d025cfe9c384ad78cadd67707d7c547eb2f1e0a1ec16aa1b"} Oct 30 23:00:06.704968 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.704763 4292 scope.go:122] "RemoveContainer" containerID="def2952053b3387bcc4fa4e520257286c6ebc2126dfdb25891bf2ef47fc602db" Oct 30 23:00:06.873448 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.873412 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:06.879686 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:06.879657 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b8bfacd-29890020-2dzk6"] Oct 30 23:00:08.146165 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.146120 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" path="/var/lib/kubelet/pods/7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba/volumes" Oct 30 23:00:08.374273 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.374234 4292 patch_prober.go:29] interesting pod/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:00:08.374443 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.374306 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:00:08.401327 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401253 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:00:08.401686 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401674 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:08.401735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401687 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:08.401735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401698 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6b115414-ceca-4273-9a16-a00d535b2af6" containerName="payload-processing" Oct 30 23:00:08.401735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401703 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b115414-ceca-4273-9a16-a00d535b2af6" containerName="payload-processing" Oct 30 23:00:08.401735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401719 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" containerName="payload-pre-processing" Oct 30 23:00:08.401735 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401725 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" containerName="payload-pre-processing" Oct 30 23:00:08.401896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401788 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:08.401896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401795 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b115414-ceca-4273-9a16-a00d535b2af6" containerName="payload-processing" Oct 30 23:00:08.401896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.401803 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3527f5aa-2d8f-4633-b7e2-416f7eec1a2f" containerName="payload-pre-processing" Oct 30 23:00:08.406157 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.406141 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.466299 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.466267 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-de0687-openshift-default-dockercfg-h9bpv\"" Oct 30 23:00:08.512723 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.512689 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:00:08.541961 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.541931 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542115 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542006 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542115 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542073 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542115 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542096 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542228 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542115 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542228 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542136 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542296 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542285 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542332 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542313 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.542367 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.542335 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzlr\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.643832 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.643795 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.643991 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.643959 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644050 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644024 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644085 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644179 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644088 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644179 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644113 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644298 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644201 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644298 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644238 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644298 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644251 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644298 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644272 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzlr\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644518 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644349 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644518 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644425 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644518 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644455 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.644923 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.644902 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.646465 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.646441 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.646605 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.646586 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.711793 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.711742 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzlr\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.711793 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.711757 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\") pod \"e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:08.719527 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:08.719505 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:09.167498 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.167465 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:00:09.203355 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:00:09.203324 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed552bc_59a5_46df_be43_1a6627005301.slice/crio-fb76d2503f4842d435e724f3cfa1dd0d5243befeefd32077b8a51af4b9561595 WatchSource:0}: Error finding container fb76d2503f4842d435e724f3cfa1dd0d5243befeefd32077b8a51af4b9561595: Status 404 returned error can't find the container with id fb76d2503f4842d435e724f3cfa1dd0d5243befeefd32077b8a51af4b9561595 Oct 30 23:00:09.205535 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.205505 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:09.205666 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.205568 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:09.205666 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.205634 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:09.730028 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.729994 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerStarted","Data":"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d"} Oct 30 23:00:09.730182 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.730072 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerStarted","Data":"fb76d2503f4842d435e724f3cfa1dd0d5243befeefd32077b8a51af4b9561595"} Oct 30 23:00:09.855268 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:09.855218 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podStartSLOduration=1.8552047040000001 podStartE2EDuration="1.855204704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:09.854228506 +0000 UTC m=+3937.823048074" watchObservedRunningTime="2026-10-30 23:00:09.855204704 +0000 UTC m=+3937.824024335" Oct 30 23:00:10.720306 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:10.720269 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:10.725085 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:10.725061 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:10.735631 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:10.735607 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:10.736362 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:10.736345 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:11.709489 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.709464 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 23:00:11.741343 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.741283 4292 generic.go:356] "Generic (PLEG): container finished" podID="28fe6107-3b32-448f-a00e-408b6304d99c" containerID="d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170" exitCode=0 Oct 30 23:00:11.741343 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.741327 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" Oct 30 23:00:11.741733 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.741362 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerDied","Data":"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170"} Oct 30 23:00:11.741733 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.741412 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474" event={"ID":"28fe6107-3b32-448f-a00e-408b6304d99c","Type":"ContainerDied","Data":"5b50dac94b6dc97bc1a86d6e852b421ab53f6fa1d7c0b63c45e28b1072076e55"} Oct 30 23:00:11.741733 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.741436 4292 scope.go:122] "RemoveContainer" containerID="d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170" Oct 30 23:00:11.750071 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.750045 4292 scope.go:122] "RemoveContainer" containerID="c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b" Oct 30 23:00:11.757691 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.757676 4292 scope.go:122] "RemoveContainer" containerID="d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170" Oct 30 23:00:11.757920 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:11.757900 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170\": container with ID starting with d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170 not found: ID does not exist" containerID="d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170" Oct 30 23:00:11.757970 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.757927 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170"} err="failed to get container status \"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170\": rpc error: code = NotFound desc = could not find container \"d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170\": container with ID starting with d95cd8f328973dcef74927a5b7937669035f8a61bb46ffd574c55cfe8e103170 not found: ID does not exist" Oct 30 23:00:11.757970 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.757944 4292 scope.go:122] "RemoveContainer" containerID="c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b" Oct 30 23:00:11.758159 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:11.758144 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b\": container with ID starting with c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b not found: ID does not exist" containerID="c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b" Oct 30 23:00:11.758205 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.758165 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b"} err="failed to get container status \"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b\": rpc error: code = NotFound desc = could not find container \"c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b\": container with ID starting with c82189acf5c8b86c29cfd13a6a813486403cd5837c48bae8263eddfa13e8314b not found: ID does not exist" Oct 30 23:00:11.876753 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876715 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.876861 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876759 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.876861 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.876861 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876811 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.876861 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876842 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.877044 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876891 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.877044 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876919 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.877044 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876955 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.877044 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876989 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\") pod \"28fe6107-3b32-448f-a00e-408b6304d99c\" (UID: \"28fe6107-3b32-448f-a00e-408b6304d99c\") " Oct 30 23:00:11.877044 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.876995 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:00:11.877320 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877117 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:00:11.877320 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877136 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:00:11.877432 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877329 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:11.877496 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877470 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.877597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877494 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.877597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877508 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.877597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877521 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fe6107-3b32-448f-a00e-408b6304d99c-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.877779 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.877727 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:00:11.879253 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.879228 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:00:11.879388 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.879316 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:00:11.879434 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.879399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:11.879434 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.879416 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6" pod "28fe6107-3b32-448f-a00e-408b6304d99c" (UID: "28fe6107-3b32-448f-a00e-408b6304d99c"). InnerVolumeSpecName "kube-api-access-skxd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:11.978084 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.978058 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.978084 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.978082 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fe6107-3b32-448f-a00e-408b6304d99c-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.978238 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.978093 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.978238 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.978102 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/28fe6107-3b32-448f-a00e-408b6304d99c-kube-api-access-skxd6\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:11.978238 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:11.978112 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fe6107-3b32-448f-a00e-408b6304d99c-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:12.167428 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:12.167396 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 23:00:12.173319 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:12.173296 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b8bfacd-openshift-default-5bc64bb98c-f5474"] Oct 30 23:00:14.145414 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:14.145378 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" path="/var/lib/kubelet/pods/28fe6107-3b32-448f-a00e-408b6304d99c/volumes" Oct 30 23:00:15.139171 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:15.139137 4292 scope.go:122] "RemoveContainer" containerID="e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c" Oct 30 23:00:15.765347 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:15.765314 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerStarted","Data":"0bf75dd86b3967e49638e41455131b36c206109b2bdd8f454ae04419d8596dcc"} Oct 30 23:00:15.891506 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:15.891459 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" podStartSLOduration=15.891447424 podStartE2EDuration="15.891447424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:15.89068638 +0000 UTC m=+3943.859505949" watchObservedRunningTime="2026-10-30 23:00:15.891447424 +0000 UTC m=+3943.860266988" Oct 30 23:00:16.561636 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.561597 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:00:16.562015 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562002 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562089 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562016 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562089 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562027 4292 cpu_manager.go:414] "containerMap: removing container" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562089 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562032 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562089 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562048 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:16.562089 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562056 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:16.562269 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562197 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562269 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562213 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fe6107-3b32-448f-a00e-408b6304d99c" containerName="istio-proxy" Oct 30 23:00:16.562269 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.562223 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5be3f0-bee8-4e4f-a64f-21aa4a38b6ba" containerName="cleanup" Oct 30 23:00:16.565351 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.565323 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.617541 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.617511 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.617700 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.617568 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmv5\" (UniqueName: \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.620122 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.620105 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-de0687\"" Oct 30 23:00:16.669702 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.669674 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-de0687-dockercfg-9wp4x\"" Oct 30 23:00:16.672950 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.672922 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:00:16.681120 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.681093 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:00:16.698988 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.698957 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.718376 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.718333 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.718547 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.718395 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmv5\" (UniqueName: \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.718547 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.718469 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5vh\" (UniqueName: \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.718547 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.718499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.719289 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.719262 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.745122 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.745091 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:00:16.773322 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.773290 4292 generic.go:356] "Generic (PLEG): container finished" podID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerID="0bf75dd86b3967e49638e41455131b36c206109b2bdd8f454ae04419d8596dcc" exitCode=60 Oct 30 23:00:16.773720 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.773314 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerDied","Data":"0bf75dd86b3967e49638e41455131b36c206109b2bdd8f454ae04419d8596dcc"} Oct 30 23:00:16.773720 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.773376 4292 scope.go:122] "RemoveContainer" containerID="e3e1092032171afd27960643c547fc55005a43cca4493babc3834850fd5aa61c" Oct 30 23:00:16.773842 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.773772 4292 scope.go:122] "RemoveContainer" containerID="0bf75dd86b3967e49638e41455131b36c206109b2bdd8f454ae04419d8596dcc" Oct 30 23:00:16.774016 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:16.773995 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-wtvp9_odh-ai-gateway-infra(86c453db-8174-4ad3-ae01-c568c6ff6e21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" Oct 30 23:00:16.783780 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.783755 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmv5\" (UniqueName: \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\") pod \"payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.819705 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.819648 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5vh\" (UniqueName: \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.819705 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.819693 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.820326 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.820308 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.877650 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.877616 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:16.882789 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.882770 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5vh\" (UniqueName: \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\") pod \"payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:16.904833 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:16.904809 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:17.018236 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.018205 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:17.259005 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.258975 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:00:17.260068 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:00:17.260037 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284b6dce_61c6_48bd_9b16_bce331a95600.slice/crio-a1ff4e231499f0cb381a4b1e20e1c0920c696cf3c9ec629fc4437afeb288743b WatchSource:0}: Error finding container a1ff4e231499f0cb381a4b1e20e1c0920c696cf3c9ec629fc4437afeb288743b: Status 404 returned error can't find the container with id a1ff4e231499f0cb381a4b1e20e1c0920c696cf3c9ec629fc4437afeb288743b Oct 30 23:00:17.400491 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.400462 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:00:17.402860 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:00:17.402837 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd554a234_0f3a_4ad6_bec1_ab92abee8f59.slice/crio-29ebb37ff3d06b42b37c4c95704a34d2423b5d1f538bab074babc5a6ef77fa9f WatchSource:0}: Error finding container 29ebb37ff3d06b42b37c4c95704a34d2423b5d1f538bab074babc5a6ef77fa9f: Status 404 returned error can't find the container with id 29ebb37ff3d06b42b37c4c95704a34d2423b5d1f538bab074babc5a6ef77fa9f Oct 30 23:00:17.781649 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.781422 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" event={"ID":"d554a234-0f3a-4ad6-bec1-ab92abee8f59","Type":"ContainerStarted","Data":"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef"} Oct 30 23:00:17.782273 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.782245 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" event={"ID":"d554a234-0f3a-4ad6-bec1-ab92abee8f59","Type":"ContainerStarted","Data":"29ebb37ff3d06b42b37c4c95704a34d2423b5d1f538bab074babc5a6ef77fa9f"} Oct 30 23:00:17.782403 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.782286 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:17.784066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.784039 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" event={"ID":"284b6dce-61c6-48bd-9b16-bce331a95600","Type":"ContainerStarted","Data":"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c"} Oct 30 23:00:17.784231 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.784090 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" event={"ID":"284b6dce-61c6-48bd-9b16-bce331a95600","Type":"ContainerStarted","Data":"a1ff4e231499f0cb381a4b1e20e1c0920c696cf3c9ec629fc4437afeb288743b"} Oct 30 23:00:17.784231 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.784199 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:17.907388 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.907323 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" podStartSLOduration=1.90730756 podStartE2EDuration="1.90730756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:17.907298084 +0000 UTC m=+3945.876117648" watchObservedRunningTime="2026-10-30 23:00:17.90730756 +0000 UTC m=+3945.876127125" Oct 30 23:00:17.942714 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:17.942687 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:18.035225 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.035141 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\") pod \"86c453db-8174-4ad3-ae01-c568c6ff6e21\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " Oct 30 23:00:18.035225 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.035200 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\" (UniqueName: \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\") pod \"86c453db-8174-4ad3-ae01-c568c6ff6e21\" (UID: \"86c453db-8174-4ad3-ae01-c568c6ff6e21\") " Oct 30 23:00:18.035533 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.035511 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca" pod "86c453db-8174-4ad3-ae01-c568c6ff6e21" (UID: "86c453db-8174-4ad3-ae01-c568c6ff6e21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:18.037347 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.037318 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj" pod "86c453db-8174-4ad3-ae01-c568c6ff6e21" (UID: "86c453db-8174-4ad3-ae01-c568c6ff6e21"). InnerVolumeSpecName "kube-api-access-h25fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:18.136066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.136038 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c453db-8174-4ad3-ae01-c568c6ff6e21-maas-api-service-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:18.136066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.136062 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h25fj\" (UniqueName: \"kubernetes.io/projected/86c453db-8174-4ad3-ae01-c568c6ff6e21-kube-api-access-h25fj\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:18.151951 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.151904 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" podStartSLOduration=2.151893715 podStartE2EDuration="2.151893715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:18.151183712 +0000 UTC m=+3946.120003285" watchObservedRunningTime="2026-10-30 23:00:18.151893715 +0000 UTC m=+3946.120713279" Oct 30 23:00:18.791963 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.791931 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" Oct 30 23:00:18.792463 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.791964 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9" event={"ID":"86c453db-8174-4ad3-ae01-c568c6ff6e21","Type":"ContainerDied","Data":"d4abefdf958e10f57406aa07c7ccf001d22617b54a2fa93a97dd20e3ea688f4e"} Oct 30 23:00:18.792463 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.792029 4292 scope.go:122] "RemoveContainer" containerID="0bf75dd86b3967e49638e41455131b36c206109b2bdd8f454ae04419d8596dcc" Oct 30 23:00:18.961124 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:18.961093 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:19.011374 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:19.011346 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-wtvp9"] Oct 30 23:00:20.145190 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:20.145146 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" path="/var/lib/kubelet/pods/86c453db-8174-4ad3-ae01-c568c6ff6e21/volumes" Oct 30 23:00:26.639306 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:26.639263 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:00:28.795217 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:28.795173 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:00:28.795705 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:28.795357 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:00:31.864110 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.864028 4292 generic.go:356] "Generic (PLEG): container finished" podID="1ed552bc-59a5-46df-be43-1a6627005301" containerID="a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d" exitCode=0 Oct 30 23:00:31.864110 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.864100 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerDied","Data":"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d"} Oct 30 23:00:31.864630 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.864416 4292 scope.go:122] "RemoveContainer" containerID="a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d" Oct 30 23:00:31.865546 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.865514 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:31.865654 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.865592 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:31.865654 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:31.865625 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:00:32.871388 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:32.871349 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerStarted","Data":"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f"} Oct 30 23:00:33.719970 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:33.719938 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:33.720202 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:33.720161 4292 patch_prober.go:29] interesting pod/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 23:00:33.720280 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:33.720216 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 23:00:34.720396 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:34.720357 4292 patch_prober.go:29] interesting pod/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 23:00:34.720827 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:34.720418 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 23:00:35.057833 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:35.057798 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm"] Oct 30 23:00:35.112066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:35.112034 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7389aaae-rsjfm"] Oct 30 23:00:35.720355 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:35.720317 4292 patch_prober.go:29] interesting pod/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 23:00:35.720560 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:35.720373 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 23:00:36.146431 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:36.146398 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2602e803-1eff-4651-9425-3d8e778883eb" path="/var/lib/kubelet/pods/2602e803-1eff-4651-9425-3d8e778883eb/volumes" Oct 30 23:00:36.720120 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:36.720081 4292 patch_prober.go:29] interesting pod/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 23:00:36.720315 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:36.720139 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 23:00:37.720589 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:37.720534 4292 patch_prober.go:29] interesting pod/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 23:00:37.721099 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:37.720603 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 23:00:38.559790 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:38.559759 4292 scope.go:122] "RemoveContainer" containerID="ea90cd02cbd647839fe8a358e189486b21a5972348d75a6ebfc9ae5cf987236a" Oct 30 23:00:38.725340 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:38.725311 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:38.725704 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:38.725683 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:38.726416 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:00:38.726394 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:00:56.638208 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:00:56.638169 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:01:26.638468 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:01:26.638429 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:01:38.597818 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:38.597781 4292 scope.go:122] "RemoveContainer" containerID="04eccb31b7abb5ba1ffa34b27711aa58aa638ec3961aaf0a42ca30aedf7b10ac" Oct 30 23:01:53.275204 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275168 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct"] Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275613 4292 cpu_manager.go:414] "containerMap: removing container" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275626 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275657 4292 cpu_manager.go:414] "containerMap: removing container" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275663 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275676 4292 cpu_manager.go:414] "containerMap: removing container" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275712 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275681 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275921 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275759 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.275921 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.275767 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:01:53.278989 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.278974 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.324641 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.324614 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.324783 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.324692 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.324783 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.324755 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9b8z\" (UniqueName: \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.338009 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.337984 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct"] Oct 30 23:01:53.385902 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.385876 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:01:53.385902 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.385894 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:01:53.386071 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.385893 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 23:01:53.386071 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.385897 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:01:53.425935 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.425906 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.425935 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.425938 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9b8z\" (UniqueName: \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.426106 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.425982 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.426543 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.426524 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.428248 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.428226 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.489101 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.489081 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9b8z\" (UniqueName: \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.591610 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.591565 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:53.965959 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:53.965928 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct"] Oct 30 23:01:53.967415 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:01:53.967387 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e0b409_55be_4ca7_909c_4c9412e082fe.slice/crio-2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa WatchSource:0}: Error finding container 2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa: Status 404 returned error can't find the container with id 2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa Oct 30 23:01:54.295739 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:54.295709 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" event={"ID":"85e0b409-55be-4ca7-909c-4c9412e082fe","Type":"ContainerStarted","Data":"2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa"} Oct 30 23:01:55.301962 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:55.301921 4292 generic.go:356] "Generic (PLEG): container finished" podID="85e0b409-55be-4ca7-909c-4c9412e082fe" containerID="973c563fe43e2150b45497a82dfd320cc543410f32ce68b93fc426dfb28dbb5d" exitCode=0 Oct 30 23:01:55.302359 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:55.302005 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" event={"ID":"85e0b409-55be-4ca7-909c-4c9412e082fe","Type":"ContainerDied","Data":"973c563fe43e2150b45497a82dfd320cc543410f32ce68b93fc426dfb28dbb5d"} Oct 30 23:01:56.440162 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.440140 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:56.450386 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.450359 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\") pod \"85e0b409-55be-4ca7-909c-4c9412e082fe\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " Oct 30 23:01:56.450506 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.450485 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\" (UniqueName: \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\") pod \"85e0b409-55be-4ca7-909c-4c9412e082fe\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " Oct 30 23:01:56.450589 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.450531 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\") pod \"85e0b409-55be-4ca7-909c-4c9412e082fe\" (UID: \"85e0b409-55be-4ca7-909c-4c9412e082fe\") " Oct 30 23:01:56.450704 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.450684 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca" pod "85e0b409-55be-4ca7-909c-4c9412e082fe" (UID: "85e0b409-55be-4ca7-909c-4c9412e082fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:01:56.450858 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.450845 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:01:56.452425 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.452403 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca" pod "85e0b409-55be-4ca7-909c-4c9412e082fe" (UID: "85e0b409-55be-4ca7-909c-4c9412e082fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:01:56.452568 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.452550 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z" pod "85e0b409-55be-4ca7-909c-4c9412e082fe" (UID: "85e0b409-55be-4ca7-909c-4c9412e082fe"). InnerVolumeSpecName "kube-api-access-g9b8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:01:56.551256 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.551231 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9b8z\" (UniqueName: \"kubernetes.io/projected/85e0b409-55be-4ca7-909c-4c9412e082fe-kube-api-access-g9b8z\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:01:56.551256 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:56.551254 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e0b409-55be-4ca7-909c-4c9412e082fe-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:01:56.639645 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:01:56.639539 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:01:57.312657 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:57.312618 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" Oct 30 23:01:57.312657 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:57.312627 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct" event={"ID":"85e0b409-55be-4ca7-909c-4c9412e082fe","Type":"ContainerDied","Data":"2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa"} Oct 30 23:01:57.312888 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:01:57.312721 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dfeb74f0b120d9179596dce0f195eb1e55d80e9f842f8ada4f0e45d18642efa" Oct 30 23:02:04.830406 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:04.830371 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:02:04.937469 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:04.937437 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:02:09.831242 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:09.831193 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" podUID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" containerName="payload-processing" containerID="cri-o://d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef" gracePeriod=25 Oct 30 23:02:09.938484 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:09.938432 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" podUID="284b6dce-61c6-48bd-9b16-bce331a95600" containerName="payload-pre-processing" containerID="cri-o://6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c" gracePeriod=25 Oct 30 23:02:10.088029 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.087967 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:02:10.169643 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.169615 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\" (UniqueName: \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\") pod \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " Oct 30 23:02:10.169806 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.169699 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\") pod \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\" (UID: \"d554a234-0f3a-4ad6-bec1-ab92abee8f59\") " Oct 30 23:02:10.170077 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.170056 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume" pod "d554a234-0f3a-4ad6-bec1-ab92abee8f59" (UID: "d554a234-0f3a-4ad6-bec1-ab92abee8f59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:10.172304 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.172271 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh" pod "d554a234-0f3a-4ad6-bec1-ab92abee8f59" (UID: "d554a234-0f3a-4ad6-bec1-ab92abee8f59"). InnerVolumeSpecName "kube-api-access-7l5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:10.187938 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.187918 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:02:10.271033 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.271004 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\" (UniqueName: \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\") pod \"284b6dce-61c6-48bd-9b16-bce331a95600\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " Oct 30 23:02:10.271168 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.271126 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\") pod \"284b6dce-61c6-48bd-9b16-bce331a95600\" (UID: \"284b6dce-61c6-48bd-9b16-bce331a95600\") " Oct 30 23:02:10.271443 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.271420 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5vh\" (UniqueName: \"kubernetes.io/projected/d554a234-0f3a-4ad6-bec1-ab92abee8f59-kube-api-access-7l5vh\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:10.271443 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.271445 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554a234-0f3a-4ad6-bec1-ab92abee8f59-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:10.271627 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.271446 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume" pod "284b6dce-61c6-48bd-9b16-bce331a95600" (UID: "284b6dce-61c6-48bd-9b16-bce331a95600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:10.273024 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.273008 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5" pod "284b6dce-61c6-48bd-9b16-bce331a95600" (UID: "284b6dce-61c6-48bd-9b16-bce331a95600"). InnerVolumeSpecName "kube-api-access-qqmv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:10.371887 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.371826 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmv5\" (UniqueName: \"kubernetes.io/projected/284b6dce-61c6-48bd-9b16-bce331a95600-kube-api-access-qqmv5\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:10.371887 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.371847 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b6dce-61c6-48bd-9b16-bce331a95600-plugins-config-volume\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:10.378802 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.378773 4292 generic.go:356] "Generic (PLEG): container finished" podID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" containerID="d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef" exitCode=0 Oct 30 23:02:10.378951 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.378817 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" Oct 30 23:02:10.378951 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.378856 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" event={"ID":"d554a234-0f3a-4ad6-bec1-ab92abee8f59","Type":"ContainerDied","Data":"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef"} Oct 30 23:02:10.378951 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.378908 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr" event={"ID":"d554a234-0f3a-4ad6-bec1-ab92abee8f59","Type":"ContainerDied","Data":"29ebb37ff3d06b42b37c4c95704a34d2423b5d1f538bab074babc5a6ef77fa9f"} Oct 30 23:02:10.378951 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.378936 4292 scope.go:122] "RemoveContainer" containerID="d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef" Oct 30 23:02:10.380214 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.380190 4292 generic.go:356] "Generic (PLEG): container finished" podID="284b6dce-61c6-48bd-9b16-bce331a95600" containerID="6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c" exitCode=0 Oct 30 23:02:10.380317 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.380237 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" Oct 30 23:02:10.380317 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.380259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" event={"ID":"284b6dce-61c6-48bd-9b16-bce331a95600","Type":"ContainerDied","Data":"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c"} Oct 30 23:02:10.380317 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.380293 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j" event={"ID":"284b6dce-61c6-48bd-9b16-bce331a95600","Type":"ContainerDied","Data":"a1ff4e231499f0cb381a4b1e20e1c0920c696cf3c9ec629fc4437afeb288743b"} Oct 30 23:02:10.388168 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.388146 4292 scope.go:122] "RemoveContainer" containerID="d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef" Oct 30 23:02:10.388459 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:10.388439 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef\": container with ID starting with d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef not found: ID does not exist" containerID="d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef" Oct 30 23:02:10.388546 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.388466 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef"} err="failed to get container status \"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef\": rpc error: code = NotFound desc = could not find container \"d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef\": container with ID starting with d6356d616b06ed55bfeb14bb19fb336224869991e37402d7ee682331768294ef not found: ID does not exist" Oct 30 23:02:10.388546 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.388484 4292 scope.go:122] "RemoveContainer" containerID="6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c" Oct 30 23:02:10.396204 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.396183 4292 scope.go:122] "RemoveContainer" containerID="6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c" Oct 30 23:02:10.396462 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:10.396439 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c\": container with ID starting with 6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c not found: ID does not exist" containerID="6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c" Oct 30 23:02:10.396551 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.396465 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c"} err="failed to get container status \"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c\": rpc error: code = NotFound desc = could not find container \"6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c\": container with ID starting with 6cfb39d08f1eeffcb3555e586bdab2cf8c702abb901250d7583b8826d4d9c89c not found: ID does not exist" Oct 30 23:02:10.547457 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.547428 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:02:10.553197 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.553173 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-de0687-6f57b6b575-7nb8j"] Oct 30 23:02:10.666935 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.666869 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:02:10.668925 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:10.668903 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-de0687-bf9c778ff-6dqhr"] Oct 30 23:02:12.146240 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:12.146205 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b6dce-61c6-48bd-9b16-bce331a95600" path="/var/lib/kubelet/pods/284b6dce-61c6-48bd-9b16-bce331a95600/volumes" Oct 30 23:02:12.146634 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:12.146595 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" path="/var/lib/kubelet/pods/d554a234-0f3a-4ad6-bec1-ab92abee8f59/volumes" Oct 30 23:02:17.375465 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:17.375424 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct"] Oct 30 23:02:17.425907 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:17.425876 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-de0687-nbmct"] Oct 30 23:02:18.145072 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:18.145038 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e0b409-55be-4ca7-909c-4c9412e082fe" path="/var/lib/kubelet/pods/85e0b409-55be-4ca7-909c-4c9412e082fe/volumes" Oct 30 23:02:24.845487 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:24.845450 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:02:24.845927 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:24.845683 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" containerID="cri-o://91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f" gracePeriod=30 Oct 30 23:02:26.640125 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:26.640084 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:02:30.093585 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.093551 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:02:30.253896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.253827 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.253896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.253864 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.253896 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.253891 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.253967 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.253984 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254069 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254099 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254104 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:02:30.254135 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254126 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254438 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254265 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:02:30.254438 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254346 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\") pod \"1ed552bc-59a5-46df-be43-1a6627005301\" (UID: \"1ed552bc-59a5-46df-be43-1a6627005301\") " Oct 30 23:02:30.254438 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254391 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:02:30.254630 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254457 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:30.254699 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254661 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:02:30.254756 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254730 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-data\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.254756 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254751 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed552bc-59a5-46df-be43-1a6627005301-istiod-ca-cert\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.254877 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254765 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-credential-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.254877 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.254779 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-socket\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.256220 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.256194 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:02:30.256324 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.256291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:30.256372 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.256326 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "kube-api-access-sfzlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:30.256372 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.256355 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo" pod "1ed552bc-59a5-46df-be43-1a6627005301" (UID: "1ed552bc-59a5-46df-be43-1a6627005301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:02:30.355857 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.355830 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-istio-token\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.355857 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.355854 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed552bc-59a5-46df-be43-1a6627005301-istio-podinfo\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.356042 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.355866 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzlr\" (UniqueName: \"kubernetes.io/projected/1ed552bc-59a5-46df-be43-1a6627005301-kube-api-access-sfzlr\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.356042 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.355875 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-workload-certs\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.356042 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.355883 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed552bc-59a5-46df-be43-1a6627005301-istio-envoy\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.477062 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.477028 4292 generic.go:356] "Generic (PLEG): container finished" podID="1ed552bc-59a5-46df-be43-1a6627005301" containerID="91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f" exitCode=0 Oct 30 23:02:30.477210 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.477077 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerDied","Data":"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f"} Oct 30 23:02:30.477210 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.477080 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" Oct 30 23:02:30.477210 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.477117 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd" event={"ID":"1ed552bc-59a5-46df-be43-1a6627005301","Type":"ContainerDied","Data":"fb76d2503f4842d435e724f3cfa1dd0d5243befeefd32077b8a51af4b9561595"} Oct 30 23:02:30.477210 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.477140 4292 scope.go:122] "RemoveContainer" containerID="91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f" Oct 30 23:02:30.486342 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.486327 4292 scope.go:122] "RemoveContainer" containerID="a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d" Oct 30 23:02:30.494272 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.494256 4292 scope.go:122] "RemoveContainer" containerID="91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f" Oct 30 23:02:30.494535 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:30.494517 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f\": container with ID starting with 91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f not found: ID does not exist" containerID="91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f" Oct 30 23:02:30.494628 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.494543 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f"} err="failed to get container status \"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f\": rpc error: code = NotFound desc = could not find container \"91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f\": container with ID starting with 91fb3c3eefaff7b63c9dedcee032be96b558e13a3b597fc673e93489634fc14f not found: ID does not exist" Oct 30 23:02:30.494628 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.494560 4292 scope.go:122] "RemoveContainer" containerID="a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d" Oct 30 23:02:30.494812 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:30.494788 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d\": container with ID starting with a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d not found: ID does not exist" containerID="a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d" Oct 30 23:02:30.494900 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.494819 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d"} err="failed to get container status \"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d\": rpc error: code = NotFound desc = could not find container \"a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d\": container with ID starting with a2f58e11978b0ce4cbbf5ecfd5f16ce08d5eeb28b490c1e22672dcaebcb5a85d not found: ID does not exist" Oct 30 23:02:30.645929 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.645899 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:02:30.696326 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:30.696297 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-de0687-openshift-default-5f84d88685-c4nbd"] Oct 30 23:02:32.146603 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:32.146557 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed552bc-59a5-46df-be43-1a6627005301" path="/var/lib/kubelet/pods/1ed552bc-59a5-46df-be43-1a6627005301/volumes" Oct 30 23:02:38.617556 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:38.617519 4292 scope.go:122] "RemoveContainer" containerID="2c9ab059f7832e9f469d83d3d4f3c303d5e0b2c41aaa02fae6ff239332243511" Oct 30 23:02:41.062531 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:41.062496 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8"] Oct 30 23:02:41.112806 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:41.112776 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-krlw8"] Oct 30 23:02:42.144960 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:42.144917 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6" path="/var/lib/kubelet/pods/3b0bbc9e-6d87-4dbf-bbd4-8e3119be38d6/volumes" Oct 30 23:02:55.556976 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.556897 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t"] Oct 30 23:02:55.557352 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557332 4292 cpu_manager.go:414] "containerMap: removing container" podUID="85e0b409-55be-4ca7-909c-4c9412e082fe" containerName="revoke-keys" Oct 30 23:02:55.557352 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557348 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e0b409-55be-4ca7-909c-4c9412e082fe" containerName="revoke-keys" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557358 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" containerName="payload-processing" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557364 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" containerName="payload-processing" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557372 4292 cpu_manager.go:414] "containerMap: removing container" podUID="284b6dce-61c6-48bd-9b16-bce331a95600" containerName="payload-pre-processing" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557378 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b6dce-61c6-48bd-9b16-bce331a95600" containerName="payload-pre-processing" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557384 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:02:55.557437 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557388 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557453 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e0b409-55be-4ca7-909c-4c9412e082fe" containerName="revoke-keys" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557464 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d554a234-0f3a-4ad6-bec1-ab92abee8f59" containerName="payload-processing" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557471 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b6dce-61c6-48bd-9b16-bce331a95600" containerName="payload-pre-processing" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557476 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557482 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c453db-8174-4ad3-ae01-c568c6ff6e21" containerName="cleanup" Oct 30 23:02:55.557682 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.557488 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:02:55.560595 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.560551 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.652563 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.652531 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k248w\" (UniqueName: \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.652722 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.652698 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.652774 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.652728 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.656039 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.656014 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:02:55.656223 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.656050 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:02:55.656223 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.656176 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 23:02:55.656223 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.656222 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:02:55.658213 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.658195 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t"] Oct 30 23:02:55.754200 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.754167 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.754334 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.754202 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.754334 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.754298 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k248w\" (UniqueName: \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.754889 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.754871 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.756566 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.756544 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.820175 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.820116 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k248w\" (UniqueName: \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\") pod \"maas-api-revoke-sub-f03629db-dcb7t\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:55.873037 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:55.873016 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:56.246022 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:56.245998 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t"] Oct 30 23:02:56.247707 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:02:56.247675 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff946b11_f277_4bfb_9ff6_e0968c3c4b56.slice/crio-1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3 WatchSource:0}: Error finding container 1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3: Status 404 returned error can't find the container with id 1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3 Oct 30 23:02:56.626412 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:56.626383 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" event={"ID":"ff946b11-f277-4bfb-9ff6-e0968c3c4b56","Type":"ContainerStarted","Data":"d1f9c8f4b1eda17a857c4cf78bf7bc395bcac807fb032327c6f36c01b35341b2"} Oct 30 23:02:56.626836 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:56.626439 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" event={"ID":"ff946b11-f277-4bfb-9ff6-e0968c3c4b56","Type":"ContainerStarted","Data":"1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3"} Oct 30 23:02:56.639378 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:02:56.639342 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:02:56.748328 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:56.748236 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" podStartSLOduration=1.7482196989999998 podStartE2EDuration="1.748219699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:56.746241826 +0000 UTC m=+4104.715061390" watchObservedRunningTime="2026-10-30 23:02:56.748219699 +0000 UTC m=+4104.717039263" Oct 30 23:02:57.632775 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:57.632737 4292 generic.go:356] "Generic (PLEG): container finished" podID="ff946b11-f277-4bfb-9ff6-e0968c3c4b56" containerID="d1f9c8f4b1eda17a857c4cf78bf7bc395bcac807fb032327c6f36c01b35341b2" exitCode=0 Oct 30 23:02:57.633170 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:57.632823 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" event={"ID":"ff946b11-f277-4bfb-9ff6-e0968c3c4b56","Type":"ContainerDied","Data":"d1f9c8f4b1eda17a857c4cf78bf7bc395bcac807fb032327c6f36c01b35341b2"} Oct 30 23:02:58.772227 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.772203 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:02:58.883276 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.883250 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\") pod \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " Oct 30 23:02:58.883448 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.883432 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\" (UniqueName: \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\") pod \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " Oct 30 23:02:58.883525 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.883482 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\") pod \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\" (UID: \"ff946b11-f277-4bfb-9ff6-e0968c3c4b56\") " Oct 30 23:02:58.883607 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.883554 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca" pod "ff946b11-f277-4bfb-9ff6-e0968c3c4b56" (UID: "ff946b11-f277-4bfb-9ff6-e0968c3c4b56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:58.883788 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.883769 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:58.885440 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.885408 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w" pod "ff946b11-f277-4bfb-9ff6-e0968c3c4b56" (UID: "ff946b11-f277-4bfb-9ff6-e0968c3c4b56"). InnerVolumeSpecName "kube-api-access-k248w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:58.885570 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.885479 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca" pod "ff946b11-f277-4bfb-9ff6-e0968c3c4b56" (UID: "ff946b11-f277-4bfb-9ff6-e0968c3c4b56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:02:58.984994 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.984961 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k248w\" (UniqueName: \"kubernetes.io/projected/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-kube-api-access-k248w\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:58.984994 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:58.984987 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff946b11-f277-4bfb-9ff6-e0968c3c4b56-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:02:59.644942 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:59.644899 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" event={"ID":"ff946b11-f277-4bfb-9ff6-e0968c3c4b56","Type":"ContainerDied","Data":"1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3"} Oct 30 23:02:59.645113 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:59.644950 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b141fb4afc9fd921b844ddd1b8e51a99685fa7e1b4267815ff82bf6db3afea3" Oct 30 23:02:59.645113 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:02:59.644919 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcb7t" Oct 30 23:03:15.056597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.056320 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p"] Oct 30 23:03:15.111597 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.110857 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-htf2p"] Oct 30 23:03:15.454809 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.454675 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z8pdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:03:15.454809 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.454755 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z8pdj" podUID="db3c85c0-bdfa-44de-9b36-6bde1bd6bba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:03:15.465703 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.465676 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-rb4gw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:03:15.465838 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:15.465719 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-rb4gw" podUID="71ff446a-b780-4219-aef2-f8e8c4bd3cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:03:16.145190 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.145154 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf" path="/var/lib/kubelet/pods/9c6fcfe5-f6ff-4740-b7c7-2d3e696c9acf/volumes" Oct 30 23:03:16.667893 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.667860 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w"] Oct 30 23:03:16.668289 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.668276 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ff946b11-f277-4bfb-9ff6-e0968c3c4b56" containerName="revoke-keys" Oct 30 23:03:16.668335 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.668290 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff946b11-f277-4bfb-9ff6-e0968c3c4b56" containerName="revoke-keys" Oct 30 23:03:16.668379 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.668334 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:03:16.668379 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.668342 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed552bc-59a5-46df-be43-1a6627005301" containerName="istio-proxy" Oct 30 23:03:16.668444 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.668402 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff946b11-f277-4bfb-9ff6-e0968c3c4b56" containerName="revoke-keys" Oct 30 23:03:16.671684 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.671664 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.731458 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.731429 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w"] Oct 30 23:03:16.771808 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.771779 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.771947 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.771815 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.771947 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.771838 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2m8\" (UniqueName: \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.779641 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.779613 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:03:16.779641 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.779619 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:03:16.779793 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.779659 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:03:16.779793 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.779731 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 23:03:16.872468 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.872436 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.872667 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.872477 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.872667 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.872501 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2m8\" (UniqueName: \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.873185 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.873160 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.874839 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.874821 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.935385 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.935335 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2m8\" (UniqueName: \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\") pod \"maas-api-revoke-sub-0165d7d2-h9x6w\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:16.984027 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:16.984005 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:17.358029 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:17.358000 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w"] Oct 30 23:03:17.359805 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:03:17.359771 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5061bd7d_9d16_4e81_9b41_bb92a6655a7b.slice/crio-0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2 WatchSource:0}: Error finding container 0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2: Status 404 returned error can't find the container with id 0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2 Oct 30 23:03:17.783368 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:17.783331 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" event={"ID":"5061bd7d-9d16-4e81-9b41-bb92a6655a7b","Type":"ContainerStarted","Data":"b1829ca4f533fb2494986db1a7aae2811f4fe6ee1615aa40e67bd0a37495f01b"} Oct 30 23:03:17.783558 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:17.783388 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" event={"ID":"5061bd7d-9d16-4e81-9b41-bb92a6655a7b","Type":"ContainerStarted","Data":"0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2"} Oct 30 23:03:17.902520 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:17.902477 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" podStartSLOduration=1.902465471 podStartE2EDuration="1.902465471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:17.90223743 +0000 UTC m=+4125.871057007" watchObservedRunningTime="2026-10-30 23:03:17.902465471 +0000 UTC m=+4125.871285035" Oct 30 23:03:18.789360 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:18.789324 4292 generic.go:356] "Generic (PLEG): container finished" podID="5061bd7d-9d16-4e81-9b41-bb92a6655a7b" containerID="b1829ca4f533fb2494986db1a7aae2811f4fe6ee1615aa40e67bd0a37495f01b" exitCode=0 Oct 30 23:03:18.789757 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:18.789405 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" event={"ID":"5061bd7d-9d16-4e81-9b41-bb92a6655a7b","Type":"ContainerDied","Data":"b1829ca4f533fb2494986db1a7aae2811f4fe6ee1615aa40e67bd0a37495f01b"} Oct 30 23:03:19.925004 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:19.924981 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:19.998404 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:19.998375 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\") pod \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " Oct 30 23:03:19.998521 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:19.998470 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\" (UniqueName: \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\") pod \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " Oct 30 23:03:19.998521 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:19.998505 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\") pod \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\" (UID: \"5061bd7d-9d16-4e81-9b41-bb92a6655a7b\") " Oct 30 23:03:19.998883 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:19.998856 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca" pod "5061bd7d-9d16-4e81-9b41-bb92a6655a7b" (UID: "5061bd7d-9d16-4e81-9b41-bb92a6655a7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:03:20.000302 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.000280 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca" pod "5061bd7d-9d16-4e81-9b41-bb92a6655a7b" (UID: "5061bd7d-9d16-4e81-9b41-bb92a6655a7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:03:20.000508 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.000492 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8" pod "5061bd7d-9d16-4e81-9b41-bb92a6655a7b" (UID: "5061bd7d-9d16-4e81-9b41-bb92a6655a7b"). InnerVolumeSpecName "kube-api-access-6v2m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:03:20.100199 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.100174 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:20.100199 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.100196 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2m8\" (UniqueName: \"kubernetes.io/projected/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-kube-api-access-6v2m8\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:20.100357 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.100207 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5061bd7d-9d16-4e81-9b41-bb92a6655a7b-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:20.800305 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.800272 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" Oct 30 23:03:20.800305 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.800289 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h9x6w" event={"ID":"5061bd7d-9d16-4e81-9b41-bb92a6655a7b","Type":"ContainerDied","Data":"0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2"} Oct 30 23:03:20.800514 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:20.800366 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6dc5632aeaf2d647712cc7a0ee700ff5e29039260105b8a6728a9b8b0972e2" Oct 30 23:03:26.638876 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:03:26.638834 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:03:28.427382 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.427347 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq"] Oct 30 23:03:28.428166 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.428139 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5061bd7d-9d16-4e81-9b41-bb92a6655a7b" containerName="revoke-keys" Oct 30 23:03:28.428269 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.428168 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5061bd7d-9d16-4e81-9b41-bb92a6655a7b" containerName="revoke-keys" Oct 30 23:03:28.428323 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.428281 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5061bd7d-9d16-4e81-9b41-bb92a6655a7b" containerName="revoke-keys" Oct 30 23:03:28.431534 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.431513 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.481201 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.481168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.481333 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.481217 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.481333 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.481309 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgpw\" (UniqueName: \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.490441 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.490414 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq"] Oct 30 23:03:28.538108 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.538085 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:03:28.538365 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.538347 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:03:28.538409 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.538353 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:03:28.538473 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.538462 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkmqn\"" Oct 30 23:03:28.582330 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.582308 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.582458 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.582336 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.582458 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.582361 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgpw\" (UniqueName: \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.583149 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.583130 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.584750 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.584729 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.646027 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.646001 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgpw\" (UniqueName: \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\") pod \"maas-api-revoke-sub-6b19338f-zqwcq\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:28.744524 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:28.744469 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:29.119832 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:29.119803 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq"] Oct 30 23:03:29.121394 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:03:29.121363 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42102e51_fc4b_42fa_aefc_b36fbe8452c6.slice/crio-db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16 WatchSource:0}: Error finding container db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16: Status 404 returned error can't find the container with id db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16 Oct 30 23:03:29.843250 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:29.843214 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" event={"ID":"42102e51-fc4b-42fa-aefc-b36fbe8452c6","Type":"ContainerStarted","Data":"75c300d2b826720168d03ef061463749171af29732f9e6cd2ff9b8505369168a"} Oct 30 23:03:29.843644 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:29.843263 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" event={"ID":"42102e51-fc4b-42fa-aefc-b36fbe8452c6","Type":"ContainerStarted","Data":"db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16"} Oct 30 23:03:29.963175 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:29.963112 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" podStartSLOduration=1.963091843 podStartE2EDuration="1.963091843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:29.962818877 +0000 UTC m=+4137.931638441" watchObservedRunningTime="2026-10-30 23:03:29.963091843 +0000 UTC m=+4137.931911410" Oct 30 23:03:30.849478 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:30.849443 4292 generic.go:356] "Generic (PLEG): container finished" podID="42102e51-fc4b-42fa-aefc-b36fbe8452c6" containerID="75c300d2b826720168d03ef061463749171af29732f9e6cd2ff9b8505369168a" exitCode=0 Oct 30 23:03:30.849913 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:30.849527 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" event={"ID":"42102e51-fc4b-42fa-aefc-b36fbe8452c6","Type":"ContainerDied","Data":"75c300d2b826720168d03ef061463749171af29732f9e6cd2ff9b8505369168a"} Oct 30 23:03:31.988477 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:31.988449 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:32.119106 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.119038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\") pod \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " Oct 30 23:03:32.119106 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.119076 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\" (UniqueName: \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\") pod \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " Oct 30 23:03:32.119279 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.119124 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\") pod \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\" (UID: \"42102e51-fc4b-42fa-aefc-b36fbe8452c6\") " Oct 30 23:03:32.119535 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.119514 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca" pod "42102e51-fc4b-42fa-aefc-b36fbe8452c6" (UID: "42102e51-fc4b-42fa-aefc-b36fbe8452c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:03:32.121145 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.121119 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca" pod "42102e51-fc4b-42fa-aefc-b36fbe8452c6" (UID: "42102e51-fc4b-42fa-aefc-b36fbe8452c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:03:32.121279 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.121190 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw" pod "42102e51-fc4b-42fa-aefc-b36fbe8452c6" (UID: "42102e51-fc4b-42fa-aefc-b36fbe8452c6"). InnerVolumeSpecName "kube-api-access-8hgpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:03:32.220732 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.220705 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-xks-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:32.220732 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.220729 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgpw\" (UniqueName: \"kubernetes.io/projected/42102e51-fc4b-42fa-aefc-b36fbe8452c6-kube-api-access-8hgpw\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:32.220872 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.220739 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42102e51-fc4b-42fa-aefc-b36fbe8452c6-maas-api-openshift-ca\") on node \"ip-10-0-142-210.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:03:32.860107 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.860077 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" Oct 30 23:03:32.860254 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.860112 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqwcq" event={"ID":"42102e51-fc4b-42fa-aefc-b36fbe8452c6","Type":"ContainerDied","Data":"db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16"} Oct 30 23:03:32.860254 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:32.860179 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4a5878e28424df8674abc1e00ae39d7765f73852506c522c7daf2a04b34f16" Oct 30 23:03:38.647260 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:38.647225 4292 scope.go:122] "RemoveContainer" containerID="9d41b8260c80d57ffcfa508a20392968282cfb115167c71f9f62ef8a832e2d12" Oct 30 23:03:43.051730 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:43.051690 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-md2fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:03:43.053088 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:43.053039 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-md2fb" podUID="27ed66a5-595c-4751-be32-3e88be1d43d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:03:48.067510 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:48.067477 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr"] Oct 30 23:03:48.117769 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:48.117739 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d62742f-44mqr"] Oct 30 23:03:48.145823 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:48.145796 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bceec1-5457-4266-8e40-b561063c0afa" path="/var/lib/kubelet/pods/c1bceec1-5457-4266-8e40-b561063c0afa/volumes" Oct 30 23:03:51.838523 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:51.838479 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-858898f5f7-hp4sf_48fe22ae-e5a2-476f-96ce-b4bbeb7fece3/manager/0.log" Oct 30 23:03:53.880600 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:53.880547 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7vm5c_516f0f5e-adba-4e83-b968-9de5e9066b0a/server/0.log" Oct 30 23:03:55.917809 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:55.917777 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-h9x6w_5061bd7d-9d16-4e81-9b41-bb92a6655a7b/revoke-keys/0.log" Oct 30 23:03:56.148954 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:56.148918 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zqwcq_42102e51-fc4b-42fa-aefc-b36fbe8452c6/revoke-keys/0.log" Oct 30 23:03:56.408814 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:56.408779 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcb7t_ff946b11-f277-4bfb-9ff6-e0968c3c4b56/revoke-keys/0.log" Oct 30 23:03:56.638524 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:03:56.638486 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:03:59.163241 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:03:59.163212 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t48f9_669ff4ad-5e46-4f46-a21b-6b6e194643a2/kuadrant-console-plugin/0.log" Oct 30 23:04:01.387466 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:01.387436 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-679b756675-b9mf7_e518fe02-f319-476c-a217-0a81262e620c/kube-auth-proxy/0.log" Oct 30 23:04:02.323242 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:02.323209 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-md2fb_27ed66a5-595c-4751-be32-3e88be1d43d1/istio-proxy/0.log" Oct 30 23:04:02.900943 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:02.900909 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-rb4gw_71ff446a-b780-4219-aef2-f8e8c4bd3cbf/istio-proxy/0.log" Oct 30 23:04:04.192307 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:04.192273 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-z8pdj_db3c85c0-bdfa-44de-9b36-6bde1bd6bba4/istio-proxy/0.log" Oct 30 23:04:04.710335 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:04.710304 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fgm2p_f5bb2c18-fe0f-4413-a332-c6d5b9db907b/payload-pre-processing/0.log" Oct 30 23:04:04.934834 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:04.934802 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:04:06.594588 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:06.594545 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-hm5xf_bc811bd0-2ab2-46b1-8d32-95cab341dcd7/main/0.log" Oct 30 23:04:07.688528 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:07.688493 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-wtllq_4241e8aa-cf0d-417e-8039-342452b62f8b/main/0.log" Oct 30 23:04:07.966907 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:07.966824 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2229k_a3c3a411-27a4-4f18-a085-593710bff7c5/main/0.log" Oct 30 23:04:18.698097 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:18.698051 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-858898f5f7-hp4sf_48fe22ae-e5a2-476f-96ce-b4bbeb7fece3/manager/0.log" Oct 30 23:04:20.376112 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:20.376085 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7vm5c_516f0f5e-adba-4e83-b968-9de5e9066b0a/server/0.log" Oct 30 23:04:22.060905 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:22.060874 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-h9x6w_5061bd7d-9d16-4e81-9b41-bb92a6655a7b/revoke-keys/0.log" Oct 30 23:04:22.290832 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:22.290800 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zqwcq_42102e51-fc4b-42fa-aefc-b36fbe8452c6/revoke-keys/0.log" Oct 30 23:04:22.516060 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:22.515978 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcb7t_ff946b11-f277-4bfb-9ff6-e0968c3c4b56/revoke-keys/0.log" Oct 30 23:04:24.890314 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:24.890287 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t48f9_669ff4ad-5e46-4f46-a21b-6b6e194643a2/kuadrant-console-plugin/0.log" Oct 30 23:04:26.638409 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:04:26.638369 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:04:26.878640 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:26.878602 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-679b756675-b9mf7_e518fe02-f319-476c-a217-0a81262e620c/kube-auth-proxy/0.log" Oct 30 23:04:27.862912 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:27.862873 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-md2fb_27ed66a5-595c-4751-be32-3e88be1d43d1/istio-proxy/0.log" Oct 30 23:04:28.414670 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:28.414643 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-rb4gw_71ff446a-b780-4219-aef2-f8e8c4bd3cbf/istio-proxy/0.log" Oct 30 23:04:29.854093 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:29.854063 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-z8pdj_db3c85c0-bdfa-44de-9b36-6bde1bd6bba4/istio-proxy/0.log" Oct 30 23:04:30.407165 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:30.407141 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fgm2p_f5bb2c18-fe0f-4413-a332-c6d5b9db907b/payload-pre-processing/0.log" Oct 30 23:04:30.624405 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:30.624372 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:04:31.679895 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:31.679863 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-hm5xf_bc811bd0-2ab2-46b1-8d32-95cab341dcd7/main/0.log" Oct 30 23:04:32.614834 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:32.614802 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-wtllq_4241e8aa-cf0d-417e-8039-342452b62f8b/main/0.log" Oct 30 23:04:32.888334 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:32.888259 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2229k_a3c3a411-27a4-4f18-a085-593710bff7c5/main/0.log" Oct 30 23:04:38.666979 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:04:38.666944 4292 scope.go:122] "RemoveContainer" containerID="9bfba70de9975e8af7cda31975af4d13b1e8b063a0f6c75a95a621f7ce1d3950" Oct 30 23:04:56.639032 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:04:56.638983 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:05:03.010887 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:03.010843 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:05:03.242068 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:03.242043 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:05:26.639116 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:05:26.639074 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:05:38.686997 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:38.686966 4292 scope.go:122] "RemoveContainer" containerID="2e47100455fbd87ea61c60aeec60420cee426085dc446277aa6edfa7b3c7075e" Oct 30 23:05:41.526042 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:41.526012 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:05:41.758447 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:41.758420 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z5c67_81f02934-7bd1-445e-96c0-0f0a740aeaef/payload-processing/0.log" Oct 30 23:05:42.137950 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.137919 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qhgmw/must-gather-w7ft8"] Oct 30 23:05:42.138308 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.138296 4292 cpu_manager.go:414] "containerMap: removing container" podUID="42102e51-fc4b-42fa-aefc-b36fbe8452c6" containerName="revoke-keys" Oct 30 23:05:42.138360 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.138310 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42102e51-fc4b-42fa-aefc-b36fbe8452c6" containerName="revoke-keys" Oct 30 23:05:42.138399 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.138393 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="42102e51-fc4b-42fa-aefc-b36fbe8452c6" containerName="revoke-keys" Oct 30 23:05:42.142886 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.142864 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.247449 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.247415 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qhgmw\"/\"openshift-service-ca.crt\"" Oct 30 23:05:42.247694 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.247443 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qhgmw\"/\"kube-root-ca.crt\"" Oct 30 23:05:42.247811 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.247633 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qhgmw\"/\"default-dockercfg-zvzwl\"" Oct 30 23:05:42.251635 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.251561 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qhgmw/must-gather-w7ft8"] Oct 30 23:05:42.267658 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.267631 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw99g\" (UniqueName: \"kubernetes.io/projected/27e17005-8bcb-44e4-8370-9f11bbb48b29-kube-api-access-zw99g\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.267873 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.267854 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e17005-8bcb-44e4-8370-9f11bbb48b29-must-gather-output\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.368429 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.368403 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw99g\" (UniqueName: \"kubernetes.io/projected/27e17005-8bcb-44e4-8370-9f11bbb48b29-kube-api-access-zw99g\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.368524 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.368498 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e17005-8bcb-44e4-8370-9f11bbb48b29-must-gather-output\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.368853 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.368835 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e17005-8bcb-44e4-8370-9f11bbb48b29-must-gather-output\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.438192 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.438136 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw99g\" (UniqueName: \"kubernetes.io/projected/27e17005-8bcb-44e4-8370-9f11bbb48b29-kube-api-access-zw99g\") pod \"must-gather-w7ft8\" (UID: \"27e17005-8bcb-44e4-8370-9f11bbb48b29\") " pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.458187 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.458166 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" Oct 30 23:05:42.843071 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.843046 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qhgmw/must-gather-w7ft8"] Oct 30 23:05:42.844773 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:05:42.844745 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e17005_8bcb_44e4_8370_9f11bbb48b29.slice/crio-1496f0b55418420f81238b71bb82d95dba048240d175a06767558dec382f94d3 WatchSource:0}: Error finding container 1496f0b55418420f81238b71bb82d95dba048240d175a06767558dec382f94d3: Status 404 returned error can't find the container with id 1496f0b55418420f81238b71bb82d95dba048240d175a06767558dec382f94d3 Oct 30 23:05:42.846774 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:42.846756 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:05:43.513233 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:43.513194 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" event={"ID":"27e17005-8bcb-44e4-8370-9f11bbb48b29","Type":"ContainerStarted","Data":"1496f0b55418420f81238b71bb82d95dba048240d175a06767558dec382f94d3"} Oct 30 23:05:44.520207 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:44.520172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" event={"ID":"27e17005-8bcb-44e4-8370-9f11bbb48b29","Type":"ContainerStarted","Data":"b9752737e78777888cdc179af081e34f32e75c522a55036057ebc3ee5aea2acb"} Oct 30 23:05:44.520207 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:44.520214 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" event={"ID":"27e17005-8bcb-44e4-8370-9f11bbb48b29","Type":"ContainerStarted","Data":"c5cc0ac059b48ab604ad3959d99491ddd37f5194f471abd7d757d041e7728071"} Oct 30 23:05:44.641763 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:44.641708 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qhgmw/must-gather-w7ft8" podStartSLOduration=1.468731325 podStartE2EDuration="2.641694511s" totalImagesPullingTime="1.172963186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:44.640549696 +0000 UTC m=+4272.609369259" watchObservedRunningTime="2026-10-30 23:05:44.641694511 +0000 UTC m=+4272.610514075" Oct 30 23:05:52.506590 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:52.506417 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zrjkd_4e8442af-fc07-44d2-aeaf-b6ad5d77dcdd/global-pull-secret-syncer/0.log" Oct 30 23:05:54.927913 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:54.927452 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mv9k2_a81c0006-0d2d-4e09-b222-99dd64093045/konnectivity-agent/0.log" Oct 30 23:05:56.643434 ip-10-0-142-210 kubenswrapper[4292]: E1030 23:05:56.643393 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xs6rg" containerName="extractor" Oct 30 23:05:56.742067 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:05:56.742034 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-210.us-west-1.compute.internal_c63e129e0cfb39daa6732f74d4a5cd8a/haproxy/0.log" Oct 30 23:06:01.385711 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.385674 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg"] Oct 30 23:06:01.393658 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.393630 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.470766 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.470733 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-sys\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.470940 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.470775 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-lib-modules\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.470940 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.470856 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-proc\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.470940 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.470899 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlpx\" (UniqueName: \"kubernetes.io/projected/3d451a30-b8d1-4e35-a545-7e615773cf90-kube-api-access-xmlpx\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.470940 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.470920 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-podres\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.481365 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.481329 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg"] Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572195 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-sys\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572245 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-lib-modules\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572277 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-proc\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572308 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlpx\" (UniqueName: \"kubernetes.io/projected/3d451a30-b8d1-4e35-a545-7e615773cf90-kube-api-access-xmlpx\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572334 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-podres\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572457 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-podres\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572513 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-sys\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.572614 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572603 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-lib-modules\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.573195 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.572651 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d451a30-b8d1-4e35-a545-7e615773cf90-proc\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.636287 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.636215 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlpx\" (UniqueName: \"kubernetes.io/projected/3d451a30-b8d1-4e35-a545-7e615773cf90-kube-api-access-xmlpx\") pod \"perf-node-gather-daemonset-xndfg\" (UID: \"3d451a30-b8d1-4e35-a545-7e615773cf90\") " pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:01.712797 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:01.712762 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:02.090315 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.090291 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg"] Oct 30 23:06:02.091906 ip-10-0-142-210 kubenswrapper[4292]: W1030 23:06:02.091870 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d451a30_b8d1_4e35_a545_7e615773cf90.slice/crio-c333fa5e7f875b189c27bcf6fe79b67a8668a71086458bca97a3b538df6b4940 WatchSource:0}: Error finding container c333fa5e7f875b189c27bcf6fe79b67a8668a71086458bca97a3b538df6b4940: Status 404 returned error can't find the container with id c333fa5e7f875b189c27bcf6fe79b67a8668a71086458bca97a3b538df6b4940 Oct 30 23:06:02.092920 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.092892 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t48f9_669ff4ad-5e46-4f46-a21b-6b6e194643a2/kuadrant-console-plugin/0.log" Oct 30 23:06:02.687705 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.687667 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" event={"ID":"3d451a30-b8d1-4e35-a545-7e615773cf90","Type":"ContainerStarted","Data":"e1447fadf05dcee114d121b201a2d284605d0234bc538324f1300175686e93e5"} Oct 30 23:06:02.688205 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.687732 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" event={"ID":"3d451a30-b8d1-4e35-a545-7e615773cf90","Type":"ContainerStarted","Data":"c333fa5e7f875b189c27bcf6fe79b67a8668a71086458bca97a3b538df6b4940"} Oct 30 23:06:02.688205 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.687897 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:02.807626 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:02.807561 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" podStartSLOduration=1.807548594 podStartE2EDuration="1.807548594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:02.806782428 +0000 UTC m=+4290.775601993" watchObservedRunningTime="2026-10-30 23:06:02.807548594 +0000 UTC m=+4290.776368157" Oct 30 23:06:08.703624 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:08.703590 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qhgmw/perf-node-gather-daemonset-xndfg" Oct 30 23:06:09.540900 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:09.540866 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2sxz4_405e081e-ac16-4ae4-8050-ba36e14b2d32/kube-state-metrics/0.log" Oct 30 23:06:09.997949 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:09.997915 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2sxz4_405e081e-ac16-4ae4-8050-ba36e14b2d32/kube-rbac-proxy-main/0.log" Oct 30 23:06:10.420066 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:10.420038 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2sxz4_405e081e-ac16-4ae4-8050-ba36e14b2d32/kube-rbac-proxy-self/0.log" Oct 30 23:06:14.343441 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:14.343405 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4nn9_be21030f-b519-4fef-9167-440ba1615ff8/node-exporter/0.log" Oct 30 23:06:14.761709 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:14.761634 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4nn9_be21030f-b519-4fef-9167-440ba1615ff8/kube-rbac-proxy/0.log" Oct 30 23:06:15.179262 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:15.179234 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4nn9_be21030f-b519-4fef-9167-440ba1615ff8/init-textfile/0.log" Oct 30 23:06:16.910530 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:16.910502 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/prometheus/0.log" Oct 30 23:06:17.327617 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:17.327547 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/config-reloader/0.log" Oct 30 23:06:17.749312 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:17.749227 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/thanos-sidecar/0.log" Oct 30 23:06:18.166523 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:18.166490 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/kube-rbac-proxy-web/0.log" Oct 30 23:06:18.585229 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:18.585202 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/kube-rbac-proxy/0.log" Oct 30 23:06:19.003164 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:19.003098 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/kube-rbac-proxy-thanos/0.log" Oct 30 23:06:19.419901 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:19.419871 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c76233e0-8e09-4f9e-8e9e-963f8462516e/init-config-reloader/0.log" Oct 30 23:06:19.861108 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:19.861077 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-598l5_a855eb26-b20b-452c-826d-73a8263ef31d/prometheus-operator/0.log" Oct 30 23:06:20.275092 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:20.274999 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-598l5_a855eb26-b20b-452c-826d-73a8263ef31d/kube-rbac-proxy/0.log" Oct 30 23:06:20.694726 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:20.694697 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6jm8q_e6fd4e3b-b4ef-4693-a404-5fa82006825b/prometheus-operator-admission-webhook/0.log" Oct 30 23:06:22.398140 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:22.398110 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/thanos-query/0.log" Oct 30 23:06:22.818569 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:22.818542 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/kube-rbac-proxy-web/0.log" Oct 30 23:06:23.235969 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:23.235887 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/kube-rbac-proxy/0.log" Oct 30 23:06:23.653685 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:23.653661 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/prom-label-proxy/0.log" Oct 30 23:06:24.073423 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:24.073400 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/kube-rbac-proxy-rules/0.log" Oct 30 23:06:24.490315 ip-10-0-142-210 kubenswrapper[4292]: I1030 23:06:24.490230 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b9668b8-xsbwg_e139ebf4-48aa-491c-b4e8-70c64845dee9/kube-rbac-proxy-metrics/0.log"